[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:26 dist-test-slave-0smx krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:26 dist-test-slave-0smx krb5kdc[399](info): set up 2 sockets
Jun 19 20:49:26 dist-test-slave-0smx krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 20:49:27.942550 392 mini_kdc.cc:121] Time spent starting KDC: real 1.557s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:28.055833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:28.134508 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:28.134733 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:28.134757 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:28.135927 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:28.135972 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:28.135985 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:28.136025 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:28.136055 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:28.137547 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:28.137784 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:28.138025 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:28.142954 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.143061 412 server_base.cc:1034] running on GCE node
W20260619 20:49:28.143052 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:28.143047 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.143409 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:28.143738 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:28.144896 412 hybrid_clock.cc:648] HybridClock initialized: now 1781902168144896 us; error 48 us; skew 500 ppm
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:28.147257 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:28.147631 412 webserver.cc:460] Webserver started at http://127.0.98.62:35423/ using document root <none> and password file <none>
I20260619 20:49:28.147837 412 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:28.147877 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:28.147964 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:28.149515 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "f1b0993f256146ab8d42ffa0dacd48c4"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "1af10b0818cb730b5e5b5fe39076bcdd"
server_key_iv: "77c5c4793f65d635fd93b5aae701b8e6"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.149914 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1b0993f256146ab8d42ffa0dacd48c4"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "1af10b0818cb730b5e5b5fe39076bcdd"
server_key_iv: "77c5c4793f65d635fd93b5aae701b8e6"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.151259 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:28.151955 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.152153 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:28.152254 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "f1b0993f256146ab8d42ffa0dacd48c4"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "1af10b0818cb730b5e5b5fe39076bcdd"
server_key_iv: "77c5c4793f65d635fd93b5aae701b8e6"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.152323 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:28.170652 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:28.172616 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:28.172789 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:28.176707 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45377
I20260619 20:49:28.176738 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45377 every 8 connection(s)
I20260619 20:49:28.177219 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:28.177932 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:28.180296 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4: Bootstrap starting.
I20260619 20:49:28.180456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 412
I20260619 20:49:28.180552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:28.180989 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:28.181331 480 log.cc:824] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:28.181384 392 external_mini_cluster.cc:1428] Setting key 30db212232e15921747175c9ba5c96f7
I20260619 20:49:28.182089 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4: No bootstrap required, opened a new log
I20260619 20:49:28.183748 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 20:49:28.183903 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:28.183949 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b0993f256146ab8d42ffa0dacd48c4, State: Initialized, Role: FOLLOWER
I20260619 20:49:28.184070 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 20:49:28.184162 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:28.184193 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:28.184262 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:28.185256 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 20:49:28.185439 480 leader_election.cc:302] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b0993f256146ab8d42ffa0dacd48c4; no voters:
I20260619 20:49:28.185590 480 leader_election.cc:288] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:28.185617 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:28.185714 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [term 1 LEADER]: Becoming Leader. State: Replica: f1b0993f256146ab8d42ffa0dacd48c4, State: Running, Role: LEADER
I20260619 20:49:28.185788 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 20:49:28.185820 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:28.186167 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b0993f256146ab8d42ffa0dacd48c4. Latest consensus state: current_term: 1 leader_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } } }
I20260619 20:49:28.186172 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0993f256146ab8d42ffa0dacd48c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } } }
I20260619 20:49:28.186345 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:28.186337 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:28.186707 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:28.187403 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:28.188931 491 catalog_manager.cc:1347] Generated new cluster ID: 6c61cb99b0c24920a49ec48a66bc195b
I20260619 20:49:28.188982 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:28.193331 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:28.193953 491 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:28.197173 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.182224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.0.98.62:45377)
0619 20:49:28.182415 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:28.182424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:28.182654 (+ 230us) server_negotiation.cc:407] Connection header received
0619 20:49:28.184267 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:28.184274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:28.184327 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:28.184474 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:28.186306 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.187847 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.189364 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.189776 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.189906 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.193082 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:28.193086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:28.193086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:28.193097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:28.195257 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.195732 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.195735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.195736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.195854 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.196120 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.196120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.196121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.196397 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:28.196564 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:28.196812 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 20:49:28.196964 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260619 20:49:28.200232 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1b0993f256146ab8d42ffa0dacd48c4: Generated new TSK 0
I20260619 20:49:28.200379 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:28.227916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:28.306844 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:28.307097 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:28.307135 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:28.308287 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:28.308352 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:28.308399 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:28.309906 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:28.310166 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:28.310387 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:28.315958 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:28.315960 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.316192 506 server_base.cc:1034] running on GCE node
W20260619 20:49:28.315960 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.316425 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:28.316695 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:28.317852 506 hybrid_clock.cc:648] HybridClock initialized: now 1781902168317840 us; error 31 us; skew 500 ppm
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:28.320263 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:28.320705 506 webserver.cc:460] Webserver started at http://127.0.98.1:44715/ using document root <none> and password file <none>
I20260619 20:49:28.320887 506 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:28.320925 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:28.321010 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:28.322016 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "8263d1019df142c4b05121bffb88ab48"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "5b0a8e16a50f0ac6c527c8bb6ad2c8d5"
server_key_iv: "049e2d1c018b97b0670da8ce843fb7bc"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.322329 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8263d1019df142c4b05121bffb88ab48"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "5b0a8e16a50f0ac6c527c8bb6ad2c8d5"
server_key_iv: "049e2d1c018b97b0670da8ce843fb7bc"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.323616 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:49:28.324326 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.324514 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:28.324601 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "8263d1019df142c4b05121bffb88ab48"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "5b0a8e16a50f0ac6c527c8bb6ad2c8d5"
server_key_iv: "049e2d1c018b97b0670da8ce843fb7bc"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.324656 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:28.340716 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:28.342928 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:28.343032 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:28.343199 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:28.343425 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:28.343453 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.343473 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:28.343494 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.349140 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45199
I20260619 20:49:28.349200 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45199 every 8 connection(s)
I20260619 20:49:28.349540 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:28.352577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 506
I20260619 20:49:28.352689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:28.352841 392 external_mini_cluster.cc:1428] Setting key 7120a43c8f2520ecef0de29140f8e2ff
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:28.359555 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.350224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44039 (local address 127.0.98.62:45377)
0619 20:49:28.350301 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:28.350303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:28.350449 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:49:28.351046 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:28.351047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:28.351059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:28.351145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:28.352370 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.352882 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.354134 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.354397 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.354463 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.356766 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:28.356767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:28.356768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:28.356775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:28.358155 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.358570 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.358572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.358572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.358632 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.358887 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.358888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.358888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.359017 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:28.359091 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:28.359365 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 20:49:28.359456 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:49:28.359680 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.350191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45377 (local address 127.0.98.1:44039)
0619 20:49:28.350342 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:28.350375 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:28.350860 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:28.351227 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:28.351246 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:28.352094 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:28.352274 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:28.352279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.352997 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.352998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:28.354035 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:28.354037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.354471 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.354471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:28.354636 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.355094 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:28.355102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:28.356674 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:28.358246 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:28.358252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:28.358254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:28.358454 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:28.358695 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:28.358696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:28.358697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:28.358811 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:28.359083 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:28.359088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:28.359221 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:49:28.359321 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 20:49:28.359470 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260619 20:49:28.360121 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45377
I20260619 20:49:28.360224 635 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:28.360411 635 heartbeater.cc:511] Master 127.0.98.62:45377 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:49:28.360945 444 ts_manager.cc:194] Registered new tserver with Master: 8263d1019df142c4b05121bffb88ab48 (127.0.98.1:45199)
I20260619 20:49:28.361671 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:28.379530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:28.458768 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:28.458966 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:28.458988 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:28.460018 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:28.460049 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:28.460074 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:28.461423 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:28.461593 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:28.461758 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:28.466838 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:28.466848 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.466852 640 server_base.cc:1034] running on GCE node
W20260619 20:49:28.466838 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.467216 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:28.467463 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:28.468611 640 hybrid_clock.cc:648] HybridClock initialized: now 1781902168468599 us; error 29 us; skew 500 ppm
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:28.471026 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:28.471438 640 webserver.cc:460] Webserver started at http://127.0.98.2:32905/ using document root <none> and password file <none>
I20260619 20:49:28.471654 640 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:28.471699 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:28.471841 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:28.472859 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "2de9aa8138504f6daa564fbd751b13d0"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "8921a809d2f343f45c17009227e616aa"
server_key_iv: "e66c1ba0d0d77dc1d86c6652baebbfbe"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.473217 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2de9aa8138504f6daa564fbd751b13d0"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "8921a809d2f343f45c17009227e616aa"
server_key_iv: "e66c1ba0d0d77dc1d86c6652baebbfbe"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.474545 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:28.475272 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.475449 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:28.475540 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "2de9aa8138504f6daa564fbd751b13d0"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "8921a809d2f343f45c17009227e616aa"
server_key_iv: "e66c1ba0d0d77dc1d86c6652baebbfbe"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.475616 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:28.501261 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:28.503135 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:28.503297 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:28.503532 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:28.503856 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:28.503911 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.503952 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:28.503976 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.510042 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45621
I20260619 20:49:28.510084 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45621 every 8 connection(s)
I20260619 20:49:28.510392 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:28.514468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 640
I20260619 20:49:28.514573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:28.514776 392 external_mini_cluster.cc:1428] Setting key a30b8223f8d969de763d2ab80dcc3c80
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:28.520561 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.511056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45377 (local address 127.0.98.2:58431)
0619 20:49:28.511209 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:28.511244 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:28.512128 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:28.512335 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:28.512338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:28.512893 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:28.513093 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:28.513097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.513841 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.513842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:28.514678 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:28.514681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.515092 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.515093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:28.515217 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.515801 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:28.515810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:28.517485 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:28.519279 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:28.519284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:28.519286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:28.519490 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:28.519710 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:28.519710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:28.519711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:28.519810 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:28.520093 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:28.520096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:28.520212 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 20:49:28.520293 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:49:28.520408 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
I20260619 20:49:28.520569 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.511106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58431 (local address 127.0.98.62:45377)
0619 20:49:28.511204 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:28.511207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:28.511719 (+ 512us) server_negotiation.cc:407] Connection header received
0619 20:49:28.512221 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:28.512222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:28.512232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:28.512285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:28.513187 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.513764 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.514767 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.515025 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.515108 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.517578 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:28.517579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:28.517579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:28.517587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:28.519200 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.519569 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.519570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.519571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.519636 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.519876 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.519877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.519878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.520023 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:28.520104 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:28.520343 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:49:28.520433 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 20:49:28.521045 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45377
I20260619 20:49:28.521126 769 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:28.521358 769 heartbeater.cc:511] Master 127.0.98.62:45377 requested a full tablet report, sending...
I20260619 20:49:28.521771 444 ts_manager.cc:194] Registered new tserver with Master: 2de9aa8138504f6daa564fbd751b13d0 (127.0.98.2:45621)
I20260619 20:49:28.522282 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58431
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:28.540171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:28.620446 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:28.620675 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:28.620702 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:28.621929 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:28.621982 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:28.622035 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:28.623618 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:28.623910 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:28.624154 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:28.629124 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:28.629140 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:28.629135 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:28.629339 774 server_base.cc:1034] running on GCE node
I20260619 20:49:28.629637 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:28.629899 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:28.631064 774 hybrid_clock.cc:648] HybridClock initialized: now 1781902168631060 us; error 43 us; skew 500 ppm
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:28.633566 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:28.634009 774 webserver.cc:460] Webserver started at http://127.0.98.3:35577/ using document root <none> and password file <none>
I20260619 20:49:28.634258 774 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:28.634326 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:28.634485 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:28.635637 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ab46cb38b8a47848ff419c23fa40600"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "95779de9715e0755b290ffed6f1b462a"
server_key_iv: "0d25a351c7eb9e79480a1c8d4b2f585b"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.636070 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ab46cb38b8a47848ff419c23fa40600"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "95779de9715e0755b290ffed6f1b462a"
server_key_iv: "0d25a351c7eb9e79480a1c8d4b2f585b"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.637639 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:28.638381 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.638569 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:28.638674 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "4ab46cb38b8a47848ff419c23fa40600"
format_stamp: "Formatted at 2026-06-19 20:49:28 on dist-test-slave-0smx"
server_key: "95779de9715e0755b290ffed6f1b462a"
server_key_iv: "0d25a351c7eb9e79480a1c8d4b2f585b"
server_key_version: "encryptionkey@0"
I20260619 20:49:28.638759 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:28.662619 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:28.664587 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:28.664762 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:28.665001 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:28.665362 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:28.665416 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.665457 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:28.665484 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:28.671821 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46019
I20260619 20:49:28.671893 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46019 every 8 connection(s)
I20260619 20:49:28.672222 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:28.674901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 774
I20260619 20:49:28.675012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:28.675156 392 external_mini_cluster.cc:1428] Setting key bf5db7c35b742d7f98bad5c745316c00
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:28.681468 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.672922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57145 (local address 127.0.98.62:45377)
0619 20:49:28.672985 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:28.672993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:28.673194 (+ 201us) server_negotiation.cc:407] Connection header received
0619 20:49:28.673746 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:28.673747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:28.673759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:28.673827 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:28.674753 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.675350 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.676523 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.676701 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.676756 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.678829 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:28.678830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:28.678830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:28.678837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:28.680167 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.680541 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.680542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.680542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.680595 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.680853 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.680855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.680855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.680978 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:28.681042 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:28.681264 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 20:49:28.681399 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:49:28.681628 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.672906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45377 (local address 127.0.98.3:57145)
0619 20:49:28.673067 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:28.673108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:28.673653 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:28.673905 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:28.673908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:28.674465 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:28.674656 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:28.674661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.675517 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.675518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:28.676452 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:28.676455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.676769 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.676770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:28.676869 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.677332 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:28.677341 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:28.678743 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:28.680248 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:28.680253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:28.680254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:28.680459 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:28.680665 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:28.680666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:28.680666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:28.680759 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:28.681041 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:28.681044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:28.681153 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 20:49:28.681232 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:49:28.681399 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260619 20:49:28.682138 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45377
I20260619 20:49:28.682219 903 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:28.682400 903 heartbeater.cc:511] Master 127.0.98.62:45377 requested a full tablet report, sending...
I20260619 20:49:28.682771 444 ts_manager.cc:194] Registered new tserver with Master: 4ab46cb38b8a47848ff419c23fa40600 (127.0.98.3:46019)
I20260619 20:49:28.683110 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57145
I20260619 20:49:28.687248 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:28.710744 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.703147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.0.98.1:45199)
0619 20:49:28.703321 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:28.703326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:28.703339 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:49:28.703370 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:28.703372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:28.703385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:28.703526 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:28.704061 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.704796 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.705628 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.705945 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.706038 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.708193 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:28.708200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:28.708200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:28.708206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:28.709703 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.710034 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.710037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.710037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.710086 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.710291 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.710292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.710292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.710411 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:28.710510 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:28.710605 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 20:49:28.710667 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
W20260619 20:49:28.711122 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56888
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:28.732657 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:28.725944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.0.98.1:45199)
0619 20:49:28.726023 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:28.726026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:28.726045 (+ 19us) server_negotiation.cc:407] Connection header received
0619 20:49:28.726126 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:28.726127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:28.726139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:28.726193 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:28.726769 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.727266 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.728045 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:28.728261 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:28.728322 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:28.730450 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:28.730451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:28.730451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:28.730457 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:28.731724 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.732026 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.732027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.732027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.732075 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:28.732218 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:28.732218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:28.732219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:28.732296 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:28.732383 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:28.732452 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 20:49:28.732511 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 20:49:28.733208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 506
I20260619 20:49:28.737749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 640
I20260619 20:49:28.743181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 774
I20260619 20:49:28.747370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 412
2026-06-19T20:49:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2387 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[917](info): set up 2 sockets
Jun 19 20:49:28 dist-test-slave-0smx krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 20:49:29.788800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:29 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:29.900810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44015
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:29.979530 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:29.979724 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:29.979745 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:29.980798 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:29.980832 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:29.980846 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:29.980865 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:29.980877 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:29.982244 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:29.982419 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:29.982573 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:29.987242 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:29.987280 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:29.987250 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:29.987368 930 server_base.cc:1034] running on GCE node
I20260619 20:49:29.987717 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:29.988014 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:29.989200 930 hybrid_clock.cc:648] HybridClock initialized: now 1781902169989181 us; error 36 us; skew 500 ppm
Jun 19 20:49:29 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:29.991510 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:29.991972 930 webserver.cc:460] Webserver started at http://127.0.98.62:44717/ using document root <none> and password file <none>
I20260619 20:49:29.992188 930 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:29.992254 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:29.992403 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:29.993485 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "48b0278aba1a403ba4e69ca7a966786c"
format_stamp: "Formatted at 2026-06-19 20:49:29 on dist-test-slave-0smx"
server_key: "d950c122836d9189071c8a314c8846a8"
server_key_iv: "903f4fd3628881f78e503fe27b44e4f4"
server_key_version: "encryptionkey@0"
I20260619 20:49:29.993832 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "48b0278aba1a403ba4e69ca7a966786c"
format_stamp: "Formatted at 2026-06-19 20:49:29 on dist-test-slave-0smx"
server_key: "d950c122836d9189071c8a314c8846a8"
server_key_iv: "903f4fd3628881f78e503fe27b44e4f4"
server_key_version: "encryptionkey@0"
I20260619 20:49:29.995137 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:29.995867 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:29.996027 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:29.996126 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "48b0278aba1a403ba4e69ca7a966786c"
format_stamp: "Formatted at 2026-06-19 20:49:29 on dist-test-slave-0smx"
server_key: "d950c122836d9189071c8a314c8846a8"
server_key_iv: "903f4fd3628881f78e503fe27b44e4f4"
server_key_version: "encryptionkey@0"
I20260619 20:49:29.996206 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:30.029173 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:30.031148 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:30.031322 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:30.035284 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44015
I20260619 20:49:30.035348 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44015 every 8 connection(s)
I20260619 20:49:30.035772 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:30.036630 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:30.038942 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c: Bootstrap starting.
I20260619 20:49:30.039477 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:30.039719 998 log.cc:824] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:30.040393 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c: No bootstrap required, opened a new log
I20260619 20:49:30.042117 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b0278aba1a403ba4e69ca7a966786c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44015 } }
I20260619 20:49:30.042261 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:30.042284 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0278aba1a403ba4e69ca7a966786c, State: Initialized, Role: FOLLOWER
I20260619 20:49:30.042425 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b0278aba1a403ba4e69ca7a966786c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44015 } }
I20260619 20:49:30.042512 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:30.042536 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:30.042567 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:30.043069 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b0278aba1a403ba4e69ca7a966786c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44015 } }
I20260619 20:49:30.043155 998 leader_election.cc:302] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b0278aba1a403ba4e69ca7a966786c; no voters:
I20260619 20:49:30.043277 998 leader_election.cc:288] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:30.043401 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:30.043591 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:30.043838 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [term 1 LEADER]: Becoming Leader. State: Replica: 48b0278aba1a403ba4e69ca7a966786c, State: Running, Role: LEADER
I20260619 20:49:30.043992 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b0278aba1a403ba4e69ca7a966786c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44015 } }
I20260619 20:49:30.044450 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b0278aba1a403ba4e69ca7a966786c. Latest consensus state: current_term: 1 leader_uuid: "48b0278aba1a403ba4e69ca7a966786c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b0278aba1a403ba4e69ca7a966786c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44015 } } }
I20260619 20:49:30.044543 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:30.044585 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b0278aba1a403ba4e69ca7a966786c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b0278aba1a403ba4e69ca7a966786c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44015 } } }
I20260619 20:49:30.044685 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:30.044772 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:30.045322 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:30.045584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 930
I20260619 20:49:30.045675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:30.045830 392 external_mini_cluster.cc:1428] Setting key f37aeb08a947bba32d36a01b66a26c82
I20260619 20:49:30.047837 1011 catalog_manager.cc:1347] Generated new cluster ID: d6eec384f1ca457ba665c99bd32e3c40
I20260619 20:49:30.047914 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:30.058993 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.046480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.0.98.62:44015)
0619 20:49:30.046640 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.046649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.046683 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:49:30.046809 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.046816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.046876 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.047021 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.047801 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.049284 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.050450 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.051122 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.051554 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.054651 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.054656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.054656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.054669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.056778 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.057223 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.057226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.057227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.057380 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.057602 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.057603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.057603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.058007 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.058526 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.058603 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.058744 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260619 20:49:30.081781 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:30.082347 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:30.089434 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b0278aba1a403ba4e69ca7a966786c: Generated new TSK 0
I20260619 20:49:30.089622 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:30.115713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44015
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:30.195144 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:30.195329 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:30.195351 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:30.196491 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:30.196530 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:30.196580 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:30.197994 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:30.198240 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:30.198459 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:30.203383 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:30.203445 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:30.203567 1024 server_base.cc:1034] running on GCE node
W20260619 20:49:30.203384 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:30.203850 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:30.204043 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:30.205178 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781902170205167 us; error 27 us; skew 500 ppm
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:30.207446 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:30.207834 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45239/ using document root <none> and password file <none>
I20260619 20:49:30.208050 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:30.208138 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:30.208282 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:30.209263 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "82666ad55fdf4e5aa4b12f3821246598"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "82486106b68034906d12326fe62904d8"
server_key_iv: "45fe087f0039f03917f2f9e9e4ad1bfd"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.209674 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "82666ad55fdf4e5aa4b12f3821246598"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "82486106b68034906d12326fe62904d8"
server_key_iv: "45fe087f0039f03917f2f9e9e4ad1bfd"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.210922 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:30.211549 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.211697 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:30.211789 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "82666ad55fdf4e5aa4b12f3821246598"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "82486106b68034906d12326fe62904d8"
server_key_iv: "45fe087f0039f03917f2f9e9e4ad1bfd"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.211865 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:30.224866 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:30.226653 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:30.226810 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:30.227074 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:30.227403 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:30.227438 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.227463 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:30.227510 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.233294 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46607
I20260619 20:49:30.233361 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46607 every 8 connection(s)
I20260619 20:49:30.233646 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:30.240304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1024
I20260619 20:49:30.240394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:30.240572 392 external_mini_cluster.cc:1428] Setting key a8624b2c9caa1eba47381845cc032ef2
I20260619 20:49:30.243237 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.234299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45627 (local address 127.0.98.62:44015)
0619 20:49:30.234362 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.234364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.234543 (+ 179us) server_negotiation.cc:407] Connection header received
0619 20:49:30.235037 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.235038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.235053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.235102 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.236033 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.236563 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.237538 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.237769 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.237851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.239973 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.239974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.239975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.239983 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.241751 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.242230 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.242231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.242231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.242291 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.242595 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.242595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.242596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.242695 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.242746 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.243056 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.243149 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:49:30.243409 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.234285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44015 (local address 127.0.98.1:45627)
0619 20:49:30.234429 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.234461 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.234925 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.235183 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.235186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.235740 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:30.235926 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.235930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.236653 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.236654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.237439 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.237441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.237860 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.237861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.237983 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.238478 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:30.238486 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:30.239864 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:30.241832 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:30.241838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:30.241841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:30.242155 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:30.242366 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:30.242367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:30.242367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:30.242533 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:30.242762 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:30.242768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:30.242911 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.243028 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.243179 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260619 20:49:30.243906 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44015
I20260619 20:49:30.244001 1153 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:30.244248 1153 heartbeater.cc:511] Master 127.0.98.62:44015 requested a full tablet report, sending...
I20260619 20:49:30.244850 962 ts_manager.cc:194] Registered new tserver with Master: 82666ad55fdf4e5aa4b12f3821246598 (127.0.98.1:46607)
I20260619 20:49:30.245852 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45627
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:30.264266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44015
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:30.343732 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:30.343989 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:30.344013 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:30.345291 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:30.345345 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:30.345372 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:30.346737 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:30.346896 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:30.347052 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:30.351845 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:30.351814 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:30.351833 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:30.351877 1158 server_base.cc:1034] running on GCE node
I20260619 20:49:30.352260 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:30.352517 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:30.353699 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781902170353698 us; error 42 us; skew 500 ppm
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:30.356067 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:30.356408 1158 webserver.cc:460] Webserver started at http://127.0.98.2:36315/ using document root <none> and password file <none>
I20260619 20:49:30.356572 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:30.356611 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:30.356693 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:30.357744 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "d0f66e43b98d49a1b67e1ccbaadff01f"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "398ffed8874b0ea7fe64f3fe1ebb1d62"
server_key_iv: "bad11405a36727a93862cf51e16803e2"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.358083 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d0f66e43b98d49a1b67e1ccbaadff01f"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "398ffed8874b0ea7fe64f3fe1ebb1d62"
server_key_iv: "bad11405a36727a93862cf51e16803e2"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.359381 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:30.360002 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.360172 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:30.360250 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "d0f66e43b98d49a1b67e1ccbaadff01f"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "398ffed8874b0ea7fe64f3fe1ebb1d62"
server_key_iv: "bad11405a36727a93862cf51e16803e2"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.360302 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:30.378857 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:30.381794 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:30.382004 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:30.382265 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:30.382613 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:30.382674 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.382711 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:30.382733 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.388267 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34389
I20260619 20:49:30.388355 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34389 every 8 connection(s)
I20260619 20:49:30.388654 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:30.397558 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.389387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46473 (local address 127.0.98.62:44015)
0619 20:49:30.389462 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.389465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.389699 (+ 234us) server_negotiation.cc:407] Connection header received
0619 20:49:30.390214 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.390216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.390227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.390331 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.391228 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.391730 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.392669 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.392971 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.393048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.395248 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.395249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.395249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.395255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.396398 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.396760 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.396760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.396761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.396806 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.397008 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.397008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.397008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.397101 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.397150 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.397426 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.397495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:49:30.397725 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.389380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44015 (local address 127.0.98.2:46473)
0619 20:49:30.389575 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.389617 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.390094 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.390386 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.390389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.390945 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:30.391143 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.391148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.391811 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.391812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.392598 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.392600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.393060 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.393061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.393221 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.393829 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:30.393837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:30.395168 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:30.396467 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:30.396472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:30.396474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:30.396696 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:30.396861 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:30.396862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:30.396862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:30.396955 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:30.397155 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:30.397159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:30.397264 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.397410 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.397531 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":118,"threads_started":1}
I20260619 20:49:30.398262 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44015
I20260619 20:49:30.398391 1287 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:30.398566 1287 heartbeater.cc:511] Master 127.0.98.62:44015 requested a full tablet report, sending...
I20260619 20:49:30.398813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1158
I20260619 20:49:30.398904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:30.398975 962 ts_manager.cc:194] Registered new tserver with Master: d0f66e43b98d49a1b67e1ccbaadff01f (127.0.98.2:34389)
I20260619 20:49:30.399092 392 external_mini_cluster.cc:1428] Setting key 13a5d4f2ad61248dd44ed9d434913748
I20260619 20:49:30.399351 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46473
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:30.422279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44015
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:30.500435 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:30.500638 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:30.500662 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:30.501818 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:30.501858 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:30.501885 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:30.503218 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:30.503388 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:30.503561 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:30.508414 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:30.508451 1292 server_base.cc:1034] running on GCE node
W20260619 20:49:30.508486 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:30.508414 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:30.508833 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:30.509066 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:30.510227 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781902170510207 us; error 35 us; skew 500 ppm
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:30.512758 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:30.513222 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41173/ using document root <none> and password file <none>
I20260619 20:49:30.513469 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:30.513540 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:30.513708 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:30.514802 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "198bd4fc528f46ff9168f7387aad96f5"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "c6801625665b89709872c651d3591ead"
server_key_iv: "16171c9c6f2a284fc953978a59a34a77"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.515198 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "198bd4fc528f46ff9168f7387aad96f5"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "c6801625665b89709872c651d3591ead"
server_key_iv: "16171c9c6f2a284fc953978a59a34a77"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.516476 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:49:30.517187 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.517407 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:30.517503 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "198bd4fc528f46ff9168f7387aad96f5"
format_stamp: "Formatted at 2026-06-19 20:49:30 on dist-test-slave-0smx"
server_key: "c6801625665b89709872c651d3591ead"
server_key_iv: "16171c9c6f2a284fc953978a59a34a77"
server_key_version: "encryptionkey@0"
I20260619 20:49:30.517596 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:30.526111 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:30.528273 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:30.528432 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:30.528632 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:30.528918 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:30.528971 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.529009 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:30.529036 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:30.534752 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36681
I20260619 20:49:30.534796 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36681 every 8 connection(s)
I20260619 20:49:30.535073 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:30.536518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1292
I20260619 20:49:30.536623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:30.536813 392 external_mini_cluster.cc:1428] Setting key ecaa3c0f4c71a35ab258ec7bf9733487
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:30.543954 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.535723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34777 (local address 127.0.98.62:44015)
0619 20:49:30.535815 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.535818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.535964 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:49:30.536498 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.536499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.536512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.536574 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.537714 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.538240 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.539173 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.539349 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.539399 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.541480 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.541481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.541481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.541487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.542586 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.543024 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.543024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.543025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.543070 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.543398 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.543398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.543399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.543501 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.543553 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.543817 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.543891 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:49:30.544178 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.535683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44015 (local address 127.0.98.3:34777)
0619 20:49:30.535855 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.535886 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.536354 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.536644 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.536647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.537418 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:30.537619 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.537624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.538342 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.538342 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.539105 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.539107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.539415 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.539416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.539506 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.539997 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:30.540004 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:30.541399 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:30.542668 (+ 1269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:30.542673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:30.542675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:30.542952 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:30.543130 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:30.543130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:30.543131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:30.543337 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:30.543562 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:30.543566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:30.543672 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.543758 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.543927 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260619 20:49:30.544592 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44015
I20260619 20:49:30.544695 1421 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:30.544903 1421 heartbeater.cc:511] Master 127.0.98.62:44015 requested a full tablet report, sending...
I20260619 20:49:30.545250 961 ts_manager.cc:194] Registered new tserver with Master: 198bd4fc528f46ff9168f7387aad96f5 (127.0.98.3:36681)
I20260619 20:49:30.545636 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34777
I20260619 20:49:30.548568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:49:30.555737 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.550123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.62:44015)
0619 20:49:30.550157 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.550159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.550283 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:49:30.550352 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.550353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.550362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.550410 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.550985 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.551430 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.552139 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.552321 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.552399 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.553574 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.553575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.553575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.553581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.554654 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.554987 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.554988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.554988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.555042 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.555233 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.555234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.555234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.555368 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.555467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.555572 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.555643 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:49:30.557662 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:30.559217 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:30.565555 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.561214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46607 (local address 127.0.0.1:52522)
0619 20:49:30.561498 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.561538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.561601 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.561912 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.561913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.561915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:30.561968 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.561970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.563359 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.563360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.564583 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.564585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.565242 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.565242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.565348 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.565362 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.565435 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.565484 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":69,"thread_start_us":51,"threads_started":1}
I20260619 20:49:30.565564 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.561348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.0.98.1:46607)
0619 20:49:30.561729 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.561736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.561750 (+ 14us) server_negotiation.cc:407] Connection header received
0619 20:49:30.561800 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.561803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.561816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.561897 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:30.562101 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.563238 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.564675 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.565155 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.565243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.565299 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.565456 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.565495 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":90,"threads_started":1}
I20260619 20:49:30.565798 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.561407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36681 (local address 127.0.0.1:32976)
0619 20:49:30.561511 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.561538 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.561601 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.561815 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.561818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.561820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:30.561887 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.561889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.563216 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.563217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.564485 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.564487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.565518 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.565519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.565586 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.565590 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.565664 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.565716 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":42,"threads_started":1}
I20260619 20:49:30.565910 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.561188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34389 (local address 127.0.0.1:49322)
0619 20:49:30.561497 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.561538 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.561602 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.562141 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.562142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.562144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:30.562192 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.562194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.563331 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.563332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.564583 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.564585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.565694 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.565695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.565755 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.565759 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.565810 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.565845 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":36,"thread_start_us":47,"threads_started":1}
I20260619 20:49:30.566104 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.561310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.0.98.2:34389)
0619 20:49:30.561909 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.561913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.561922 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:30.561960 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.561961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.561976 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.562046 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:30.562375 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.563228 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.564655 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.565374 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.565436 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.565484 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.565830 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.566034 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":85,"threads_started":1}
I20260619 20:49:30.566138 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.561498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.0.98.3:36681)
0619 20:49:30.561663 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.561667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.561677 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:30.561718 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.561720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.561731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.561791 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:30.561980 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.563124 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.564589 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.565348 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.565424 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.565477 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.566026 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.566065 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260619 20:49:30.566511 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4023c0f191814ca48be319122bd9a9cd (DEFAULT_TABLE table=test-table [id=e1eff08df2d444908d547a496894f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:30.566763 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 4023c0f191814ca48be319122bd9a9cd (DEFAULT_TABLE table=test-table [id=e1eff08df2d444908d547a496894f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:30.566875 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4023c0f191814ca48be319122bd9a9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:30.566825 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4023c0f191814ca48be319122bd9a9cd (DEFAULT_TABLE table=test-table [id=e1eff08df2d444908d547a496894f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:30.567054 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4023c0f191814ca48be319122bd9a9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:30.567152 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4023c0f191814ca48be319122bd9a9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:30.570101 1435 tablet_bootstrap.cc:492] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: Bootstrap starting.
I20260619 20:49:30.570190 1437 tablet_bootstrap.cc:492] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: Bootstrap starting.
I20260619 20:49:30.570713 1435 tablet_bootstrap.cc:654] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:30.570741 1437 tablet_bootstrap.cc:654] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:30.571194 1437 log.cc:824] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:30.571197 1435 log.cc:824] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:30.571523 1436 tablet_bootstrap.cc:492] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: Bootstrap starting.
I20260619 20:49:30.571915 1437 tablet_bootstrap.cc:492] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: No bootstrap required, opened a new log
I20260619 20:49:30.572007 1437 ts_tablet_manager.cc:1397] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:30.572118 1436 tablet_bootstrap.cc:654] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:30.572605 1436 log.cc:824] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:30.573427 1436 tablet_bootstrap.cc:492] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: No bootstrap required, opened a new log
I20260619 20:49:30.573480 1436 ts_tablet_manager.cc:1397] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:49:30.573918 1437 raft_consensus.cc:348] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.574084 1437 raft_consensus.cc:374] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:30.574128 1437 raft_consensus.cc:729] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198bd4fc528f46ff9168f7387aad96f5, State: Initialized, Role: FOLLOWER
I20260619 20:49:30.574241 1437 consensus_queue.cc:260] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.574494 1421 heartbeater.cc:503] Master 127.0.98.62:44015 was elected leader, sending a full tablet report...
I20260619 20:49:30.574494 1437 ts_tablet_manager.cc:1428] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:49:30.575351 1435 tablet_bootstrap.cc:492] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: No bootstrap required, opened a new log
I20260619 20:49:30.575423 1435 ts_tablet_manager.cc:1397] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 20:49:30.575587 1436 raft_consensus.cc:348] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.575768 1436 raft_consensus.cc:374] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:30.575835 1436 raft_consensus.cc:729] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82666ad55fdf4e5aa4b12f3821246598, State: Initialized, Role: FOLLOWER
I20260619 20:49:30.575948 1436 consensus_queue.cc:260] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.576215 1436 ts_tablet_manager.cc:1428] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:49:30.576251 1153 heartbeater.cc:503] Master 127.0.98.62:44015 was elected leader, sending a full tablet report...
I20260619 20:49:30.577270 1435 raft_consensus.cc:348] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.577497 1435 raft_consensus.cc:374] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:30.577544 1435 raft_consensus.cc:729] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f66e43b98d49a1b67e1ccbaadff01f, State: Initialized, Role: FOLLOWER
I20260619 20:49:30.577662 1435 consensus_queue.cc:260] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.577884 1435 ts_tablet_manager.cc:1428] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:49:30.577965 1287 heartbeater.cc:503] Master 127.0.98.62:44015 was elected leader, sending a full tablet report...
W20260619 20:49:30.639185 1288 tablet.cc:2369] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:30.716698 1441 raft_consensus.cc:482] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:30.716816 1441 raft_consensus.cc:504] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.717150 1441 leader_election.cc:288] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0f66e43b98d49a1b67e1ccbaadff01f (127.0.98.2:34389), 82666ad55fdf4e5aa4b12f3821246598 (127.0.98.1:46607)
I20260619 20:49:30.719888 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.717392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36229 (local address 127.0.98.2:34389)
0619 20:49:30.717448 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.717451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.717460 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:30.717489 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.717490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.717502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.717571 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:30.717747 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.718218 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.719122 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.719499 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.719585 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.719595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.719781 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.719825 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:49:30.719888 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.717289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34389 (local address 127.0.98.3:36229)
0619 20:49:30.717355 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.717368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.717409 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.717588 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.717589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.717591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:30.717660 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.717662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.718304 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.718305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.719031 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.719033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.719592 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.719593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.719701 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.719705 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.719775 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.719815 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260619 20:49:30.720328 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.717459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51265 (local address 127.0.98.1:46607)
0619 20:49:30.717543 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.717545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.717709 (+ 164us) server_negotiation.cc:407] Connection header received
0619 20:49:30.717773 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.717773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.717785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.717865 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:30.718021 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.718482 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.719369 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.719820 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.719907 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.719918 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.720240 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.720276 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 20:49:30.720348 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.717321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46607 (local address 127.0.98.3:51265)
0619 20:49:30.717644 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:30.717653 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:30.717722 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:30.717869 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:30.717870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:30.717871 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:30.717944 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.717945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.718561 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.718562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.719284 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:30.719285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.719910 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.719912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:30.720158 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.720163 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:30.720223 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:49:30.720260 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260619 20:49:30.720309 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4023c0f191814ca48be319122bd9a9cd" candidate_uuid: "198bd4fc528f46ff9168f7387aad96f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" is_pre_election: true
I20260619 20:49:30.720476 1241 raft_consensus.cc:2454] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198bd4fc528f46ff9168f7387aad96f5 in term 0.
I20260619 20:49:30.720688 1308 leader_election.cc:302] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198bd4fc528f46ff9168f7387aad96f5, d0f66e43b98d49a1b67e1ccbaadff01f; no voters:
I20260619 20:49:30.720745 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4023c0f191814ca48be319122bd9a9cd" candidate_uuid: "198bd4fc528f46ff9168f7387aad96f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82666ad55fdf4e5aa4b12f3821246598" is_pre_election: true
I20260619 20:49:30.720837 1441 raft_consensus.cc:2781] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:30.720886 1441 raft_consensus.cc:482] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:30.720901 1107 raft_consensus.cc:2454] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198bd4fc528f46ff9168f7387aad96f5 in term 0.
I20260619 20:49:30.720916 1441 raft_consensus.cc:3037] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:30.721622 1441 raft_consensus.cc:504] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.721823 1441 leader_election.cc:288] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [CANDIDATE]: Term 1 election: Requested vote from peers d0f66e43b98d49a1b67e1ccbaadff01f (127.0.98.2:34389), 82666ad55fdf4e5aa4b12f3821246598 (127.0.98.1:46607)
I20260619 20:49:30.721881 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4023c0f191814ca48be319122bd9a9cd" candidate_uuid: "198bd4fc528f46ff9168f7387aad96f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f"
I20260619 20:49:30.721915 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4023c0f191814ca48be319122bd9a9cd" candidate_uuid: "198bd4fc528f46ff9168f7387aad96f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82666ad55fdf4e5aa4b12f3821246598"
I20260619 20:49:30.721985 1107 raft_consensus.cc:3037] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:30.721969 1241 raft_consensus.cc:3037] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:30.722658 1107 raft_consensus.cc:2454] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198bd4fc528f46ff9168f7387aad96f5 in term 1.
I20260619 20:49:30.722671 1241 raft_consensus.cc:2454] T 4023c0f191814ca48be319122bd9a9cd P d0f66e43b98d49a1b67e1ccbaadff01f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 198bd4fc528f46ff9168f7387aad96f5 in term 1.
I20260619 20:49:30.722827 1309 leader_election.cc:302] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198bd4fc528f46ff9168f7387aad96f5, 82666ad55fdf4e5aa4b12f3821246598; no voters:
I20260619 20:49:30.722923 1441 raft_consensus.cc:2781] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:30.723075 1441 raft_consensus.cc:686] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [term 1 LEADER]: Becoming Leader. State: Replica: 198bd4fc528f46ff9168f7387aad96f5, State: Running, Role: LEADER
I20260619 20:49:30.723162 1441 consensus_queue.cc:237] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } }
I20260619 20:49:30.723907 961 catalog_manager.cc:5495] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 198bd4fc528f46ff9168f7387aad96f5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "198bd4fc528f46ff9168f7387aad96f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f66e43b98d49a1b67e1ccbaadff01f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82666ad55fdf4e5aa4b12f3821246598" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "198bd4fc528f46ff9168f7387aad96f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36681 } health_report { overall_health: HEALTHY } } }
W20260619 20:49:30.734295 1154 tablet.cc:2369] T 4023c0f191814ca48be319122bd9a9cd P 82666ad55fdf4e5aa4b12f3821246598: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:30.753607 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.746545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.0.98.1:46607)
0619 20:49:30.746601 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.746604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.746747 (+ 143us) server_negotiation.cc:407] Connection header received
0619 20:49:30.746816 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.746817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.746832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.746899 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.747489 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.747960 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.748680 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.748872 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.748956 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.751247 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.751256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.751256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.751262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.752633 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.752932 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.752935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.752935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.752997 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.753154 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.753155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.753155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.753275 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.753440 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.753489 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.753546 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:30.776096 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.769368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.0.98.1:46607)
0619 20:49:30.769432 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.769434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.769543 (+ 109us) server_negotiation.cc:407] Connection header received
0619 20:49:30.769619 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.769620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.769629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.769683 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.770188 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.770671 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.771375 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.771572 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.771645 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.773755 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.773756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.773756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.773764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.774949 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.775288 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.775289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.775290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.775345 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.775546 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.775547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.775548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.775643 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.775751 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.775927 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.775998 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260619 20:49:30.785555 1422 tablet.cc:2369] T 4023c0f191814ca48be319122bd9a9cd P 198bd4fc528f46ff9168f7387aad96f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:30.797096 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:30.790389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.98.1:46607)
0619 20:49:30.790468 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:30.790471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:30.790589 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:49:30.790631 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:30.790632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:30.790642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:30.790700 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:30.791213 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.791687 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.792379 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:30.792564 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:30.792629 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:30.794821 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:30.794823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:30.794823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:30.794829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:30.796208 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.796521 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.796522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.796522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.796569 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:30.796708 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:30.796708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:30.796708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:30.796786 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:30.796851 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:30.796956 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 20:49:30.797008 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 20:49:30.797581 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4023c0f191814ca48be319122bd9a9cd: Uninitialized: clean time has not yet been initialized
I20260619 20:49:30.798511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1024
I20260619 20:49:30.803367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1158
I20260619 20:49:30.807852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1292
I20260619 20:49:30.812057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 930
2026-06-19T20:49:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2065 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[1459](info): set up 2 sockets
Jun 19 20:49:30 dist-test-slave-0smx krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260619 20:49:31.844868 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:31 dist-test-slave-0smx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:31.957626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:32.037108 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:32.037336 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:32.037360 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:32.038527 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:32.038568 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:49:32.038581 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:32.038594 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:32.038607 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:32.038661 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:32.040126 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:32.040381 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:32.040613 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:32.045404 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.045459 1472 server_base.cc:1034] running on GCE node
W20260619 20:49:32.045414 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:32.045429 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.045861 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:32.046151 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:32.047290 1472 hybrid_clock.cc:648] HybridClock initialized: now 1781902172047269 us; error 35 us; skew 500 ppm
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:32.053424 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:32.054018 1472 webserver.cc:460] Webserver started at http://127.0.98.62:39737/ using document root <none> and password file <none>
I20260619 20:49:32.054275 1472 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:32.054363 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:32.054518 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:32.055998 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "bd2f86d25d1b406d815339f44dceb387"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "7e60231a91b3e4ce894234af7c39d36e"
server_key_iv: "0a62168439d088ad340644221de98b77"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.056445 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd2f86d25d1b406d815339f44dceb387"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "7e60231a91b3e4ce894234af7c39d36e"
server_key_iv: "0a62168439d088ad340644221de98b77"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.058332 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 20:49:32.059345 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.059602 1472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:32.059698 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "bd2f86d25d1b406d815339f44dceb387"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "7e60231a91b3e4ce894234af7c39d36e"
server_key_iv: "0a62168439d088ad340644221de98b77"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.059788 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:32.072335 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:32.074709 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:32.074867 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:32.079267 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46487
I20260619 20:49:32.079337 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46487 every 8 connection(s)
I20260619 20:49:32.079674 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:32.080353 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.081789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1472
I20260619 20:49:32.081885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:32.082041 392 external_mini_cluster.cc:1428] Setting key 544a0930bb99cee4a3681e855613f944
I20260619 20:49:32.083067 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: Bootstrap starting.
I20260619 20:49:32.083791 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.084141 1550 log.cc:824] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:32.084816 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: No bootstrap required, opened a new log
I20260619 20:49:32.087093 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2f86d25d1b406d815339f44dceb387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260619 20:49:32.087286 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.087332 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2f86d25d1b406d815339f44dceb387, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.087469 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2f86d25d1b406d815339f44dceb387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260619 20:49:32.087556 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:32.087591 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:32.087641 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.088367 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2f86d25d1b406d815339f44dceb387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260619 20:49:32.088474 1550 leader_election.cc:302] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2f86d25d1b406d815339f44dceb387; no voters:
I20260619 20:49:32.088649 1550 leader_election.cc:288] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:32.088676 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:32.088788 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [term 1 LEADER]: Becoming Leader. State: Replica: bd2f86d25d1b406d815339f44dceb387, State: Running, Role: LEADER
I20260619 20:49:32.088881 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2f86d25d1b406d815339f44dceb387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } }
I20260619 20:49:32.088907 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:32.089174 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2f86d25d1b406d815339f44dceb387. Latest consensus state: current_term: 1 leader_uuid: "bd2f86d25d1b406d815339f44dceb387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2f86d25d1b406d815339f44dceb387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } } }
I20260619 20:49:32.089257 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:32.089186 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2f86d25d1b406d815339f44dceb387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2f86d25d1b406d815339f44dceb387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46487 } } }
I20260619 20:49:32.089453 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:32.089465 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:32.090031 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:32.091635 1559 catalog_manager.cc:1347] Generated new cluster ID: 91a4a7022273473893fac9e24ff14484
I20260619 20:49:32.091694 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:32.093393 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.082529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.0.98.62:46487)
0619 20:49:32.082703 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.082711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.082735 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:49:32.083131 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.083139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.083208 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.083320 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.084133 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.085134 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.086285 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.086682 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.086768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.089656 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.089660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.089661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.089672 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.091773 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.092157 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.092160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.092161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.092261 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.092463 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.092464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.092464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.092754 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.092906 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.093010 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.093142 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260619 20:49:32.114166 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:32.114886 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:32.128952 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: Generated new TSK 0
I20260619 20:49:32.129150 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:32.164222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:32.244736 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:32.244918 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:32.244941 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:32.246140 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:32.246178 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:32.246198 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:49:32.246249 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:32.247675 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:32.247947 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:32.248193 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:32.252986 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:32.252993 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.253188 1576 server_base.cc:1034] running on GCE node
W20260619 20:49:32.253037 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.253468 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:32.253710 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:32.254848 1576 hybrid_clock.cc:648] HybridClock initialized: now 1781902172254837 us; error 28 us; skew 500 ppm
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:32.257238 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:32.257676 1576 webserver.cc:460] Webserver started at http://127.0.98.1:41213/ using document root <none> and password file <none>
I20260619 20:49:32.257912 1576 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:32.257982 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:32.258153 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:32.259195 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "4fe3924cd0b148a3b1e3e461e5a01464"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "ffe77713b81bcf1faac550815db4198d"
server_key_iv: "513efa356366fb88d227a1484742e41e"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.259573 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4fe3924cd0b148a3b1e3e461e5a01464"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "ffe77713b81bcf1faac550815db4198d"
server_key_iv: "513efa356366fb88d227a1484742e41e"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.260867 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:49:32.261598 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.261744 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:32.261847 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "4fe3924cd0b148a3b1e3e461e5a01464"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "ffe77713b81bcf1faac550815db4198d"
server_key_iv: "513efa356366fb88d227a1484742e41e"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.261924 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:32.278505 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:32.280292 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:32.280452 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:32.280931 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:32.280993 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.281039 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:32.281085 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:32.287854 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37103
I20260619 20:49:32.287961 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37103 every 8 connection(s)
I20260619 20:49:32.288237 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:32.288558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1576
I20260619 20:49:32.288642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:32.288844 392 external_mini_cluster.cc:1428] Setting key d5cd5d399231e53580ef7aab779e33a7
I20260619 20:49:32.291731 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.281466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47569 (local address 127.0.98.62:46487)
0619 20:49:32.281527 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.281529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.281714 (+ 185us) server_negotiation.cc:407] Connection header received
0619 20:49:32.282252 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.282254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.282267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.282323 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.282970 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.283777 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.285439 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.285771 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.285847 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.287976 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.287978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.287978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.287985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.290236 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.290694 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.290694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.290695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.290762 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.291033 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.291035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.291035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.291184 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.291256 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.291526 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.291615 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:49:32.291776 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.281455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.1:47569)
0619 20:49:32.281606 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.281638 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.282133 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.282389 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.282392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.282687 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:32.282876 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.282880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.283905 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.283906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.285355 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.285358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.285869 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.285871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.286010 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.286492 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:32.286501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:32.287817 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:32.290313 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.290319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.290321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.290617 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.290823 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.290824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.290824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.290961 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.291257 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:32.291261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:32.291367 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.291485 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.291626 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260619 20:49:32.292237 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46487
I20260619 20:49:32.292313 1707 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:32.292484 1707 heartbeater.cc:511] Master 127.0.98.62:46487 requested a full tablet report, sending...
I20260619 20:49:32.292996 1503 ts_manager.cc:194] Registered new tserver with Master: 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103)
I20260619 20:49:32.293746 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47569
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 20:49:32.302325 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.293284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53389 (local address 127.0.98.62:46487)
0619 20:49:32.293334 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.293336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.297114 (+ 3778us) server_negotiation.cc:407] Connection header received
0619 20:49:32.297139 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.297140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.297148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.297199 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.297540 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.298008 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.298661 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.298837 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.298887 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.299847 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.299849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.299849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.299856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.301211 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.301651 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.301652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.301652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.301698 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.301898 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.301898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.301899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.301987 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.302033 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.302192 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.302265 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:49:32.302436 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.293226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.1:53389)
0619 20:49:32.297041 (+ 3815us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.297052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.297107 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.297217 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.297218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.297407 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:32.297467 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.297481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.298081 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.298082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.298609 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.298611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.298895 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.298896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.298937 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.299415 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:32.299420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:32.299764 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:32.301338 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.301339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.301339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.301580 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.301765 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.301765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.301766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.301834 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.302045 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:32.302046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:32.302093 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.302156 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.302266 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3793,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:32.313957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:32.394709 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:32.394924 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:32.394960 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:32.396066 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:32.396099 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:32.396119 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:49:32.396135 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:32.397612 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:32.397840 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:32.398010 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:32.402822 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:32.402807 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.402912 1716 server_base.cc:1034] running on GCE node
W20260619 20:49:32.402827 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.403264 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:32.403507 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:32.404659 1716 hybrid_clock.cc:648] HybridClock initialized: now 1781902172404666 us; error 43 us; skew 500 ppm
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:32.406909 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:32.407377 1716 webserver.cc:460] Webserver started at http://127.0.98.2:40733/ using document root <none> and password file <none>
I20260619 20:49:32.407593 1716 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:32.407660 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:32.407820 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:32.408862 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e8759920e944186b4acc4b47dd89698"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "9853a8e3ce7ecb5ad2acc564864383de"
server_key_iv: "4519402cd27c6ebae0d0f06a0dc4b64e"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.409260 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8759920e944186b4acc4b47dd89698"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "9853a8e3ce7ecb5ad2acc564864383de"
server_key_iv: "4519402cd27c6ebae0d0f06a0dc4b64e"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.410557 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:32.411218 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.411386 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:32.411480 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "3e8759920e944186b4acc4b47dd89698"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "9853a8e3ce7ecb5ad2acc564864383de"
server_key_iv: "4519402cd27c6ebae0d0f06a0dc4b64e"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.411556 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:32.452586 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:32.454813 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:32.454978 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:32.455562 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:32.455626 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.455660 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:32.455703 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:32.462945 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37077
I20260619 20:49:32.463027 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37077 every 8 connection(s)
I20260619 20:49:32.463376 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:32.466111 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.456055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44413 (local address 127.0.98.62:46487)
0619 20:49:32.456122 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.456124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.456333 (+ 209us) server_negotiation.cc:407] Connection header received
0619 20:49:32.456851 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.456853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.456866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.456940 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.457754 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.458354 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.459518 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.459756 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.459843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.463268 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.463269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.463270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.463277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.464664 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.465071 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.465072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.465072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.465136 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.465444 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.465445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.465446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.465596 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.465661 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.465943 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.466022 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:49:32.466295 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.456037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.2:44413)
0619 20:49:32.456200 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.456241 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.456756 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.457055 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.457060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.457435 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:32.457641 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.457646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.458460 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.458462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.459420 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.459423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.459852 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.459853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.460005 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.460527 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:32.460539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:32.463147 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:32.464756 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.464762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.464764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.464990 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.465201 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.465202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.465203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.465368 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.465652 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:32.465658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:32.465797 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.465907 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.466065 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260619 20:49:32.466753 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46487
I20260619 20:49:32.466835 1847 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:32.467031 1847 heartbeater.cc:511] Master 127.0.98.62:46487 requested a full tablet report, sending...
I20260619 20:49:32.467432 1504 ts_manager.cc:194] Registered new tserver with Master: 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077)
I20260619 20:49:32.467867 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44413
I20260619 20:49:32.468904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1716
I20260619 20:49:32.468991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:32.469166 392 external_mini_cluster.cc:1428] Setting key b27982c9e454e170f886ef4eac69a9f4
I20260619 20:49:32.473884 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.467769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.2:37697)
0619 20:49:32.467895 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.467905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.467962 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.468087 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.468088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.468285 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:32.468363 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.468365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.469109 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.469110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.469918 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.469920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.470381 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.470381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.470429 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.470865 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:32.470869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:32.471179 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:32.472784 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.472785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.472785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.472942 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.473164 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.473165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.473165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.473216 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.473575 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:32.473576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:32.473610 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.473711 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.473811 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260619 20:49:32.473968 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.467827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37697 (local address 127.0.98.62:46487)
0619 20:49:32.467890 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.467892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.467948 (+ 56us) server_negotiation.cc:407] Connection header received
0619 20:49:32.468018 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.468018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.468028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.468082 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.468442 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.469023 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.469992 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.470307 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.470379 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.471273 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.471275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.471275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.471281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.472708 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.473015 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.473016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.473016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.473077 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.473280 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.473281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.473281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.473505 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.473581 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.473755 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.473840 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:32.494235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46487
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:32.574227 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:32.574493 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:32.574532 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:32.575717 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:32.575786 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:32.575824 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:49:32.575881 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:32.577425 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:32.577684 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:32.577907 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:32.582705 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:32.582702 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.582844 1856 server_base.cc:1034] running on GCE node
W20260619 20:49:32.582877 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:32.583154 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:32.583412 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:32.584584 1856 hybrid_clock.cc:648] HybridClock initialized: now 1781902172584573 us; error 24 us; skew 500 ppm
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:32.587004 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:32.587450 1856 webserver.cc:460] Webserver started at http://127.0.98.3:32809/ using document root <none> and password file <none>
I20260619 20:49:32.587636 1856 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:32.587679 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:32.587766 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:32.588857 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "99cca9446d5b496f9582c11dc7316e81"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "c2a33136b9c73e584205e466801f1230"
server_key_iv: "9e63bced93f5b9d09e8e755fe198cc92"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.589195 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "99cca9446d5b496f9582c11dc7316e81"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "c2a33136b9c73e584205e466801f1230"
server_key_iv: "9e63bced93f5b9d09e8e755fe198cc92"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.590667 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:32.591398 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.591554 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:32.591653 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "99cca9446d5b496f9582c11dc7316e81"
format_stamp: "Formatted at 2026-06-19 20:49:32 on dist-test-slave-0smx"
server_key: "c2a33136b9c73e584205e466801f1230"
server_key_iv: "9e63bced93f5b9d09e8e755fe198cc92"
server_key_version: "encryptionkey@0"
I20260619 20:49:32.591743 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:32.605454 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:32.607726 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:32.607900 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:32.608372 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:32.608431 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.608475 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:32.608536 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.615013 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36333
I20260619 20:49:32.615095 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36333 every 8 connection(s)
I20260619 20:49:32.615423 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:32.618696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 1856
I20260619 20:49:32.618801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:32.618995 392 external_mini_cluster.cc:1428] Setting key e8891b1c93ed1472682fce4caa35381a
I20260619 20:49:32.619321 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.608952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42693 (local address 127.0.98.62:46487)
0619 20:49:32.609024 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.609026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.609266 (+ 240us) server_negotiation.cc:407] Connection header received
0619 20:49:32.609864 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.609866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.609879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.609983 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.610865 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.611726 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.613316 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.613669 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.613757 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.616418 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.616419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.616419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.616425 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.617885 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.618233 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.618234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.618234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.618291 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.618581 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.618582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.618583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.618722 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.618803 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.619192 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.619252 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:49:32.619580 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.608911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.3:42693)
0619 20:49:32.609094 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.609164 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.609724 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.610059 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.610063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.610444 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:32.610750 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.610758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.611830 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.611831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.613186 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.613190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.613768 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.613769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.613925 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.614571 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:32.614587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:32.616339 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:32.617957 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.617962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.617963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.618169 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.618376 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.618377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.618377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.618488 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.618866 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:32.618877 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:32.619035 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.619152 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.619331 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260619 20:49:32.620034 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46487
I20260619 20:49:32.620111 1987 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:32.620296 1987 heartbeater.cc:511] Master 127.0.98.62:46487 requested a full tablet report, sending...
I20260619 20:49:32.620644 1503 ts_manager.cc:194] Registered new tserver with Master: 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333)
I20260619 20:49:32.621023 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42693
I20260619 20:49:32.626048 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.621108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47247 (local address 127.0.98.62:46487)
0619 20:49:32.621231 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.621233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.621241 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:32.621316 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.621317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.621327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.621385 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.621733 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.622168 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.622797 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.622969 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.623020 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.623770 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.623771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.623772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.623777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.625147 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.625508 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.625508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.625509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.625551 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.625706 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.625707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.625707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.625792 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.625835 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.625939 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.625990 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 20:49:32.626102 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.621046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.98.3:47247)
0619 20:49:32.621152 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.621162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.621251 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.621400 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.621400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.621602 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:32.621664 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.621665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.622249 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.622250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.622728 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.622729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.623042 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.623042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.623093 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.623424 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:32.623428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:32.623694 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:32.625222 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.625223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.625223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.625450 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.625616 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:32.625617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:32.625617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:32.625671 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:32.625840 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:32.625841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:32.625872 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.625925 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.625996 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260619 20:49:32.630774 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:49:32 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:32.662068 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.654318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.0.98.62:46487)
0619 20:49:32.654361 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.654363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.654371 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:32.654420 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.654421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.654430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.654503 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:32.655355 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.656216 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.657279 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.657533 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.657588 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.659941 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:32.659942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:32.659942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:32.659949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:32.661096 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.661459 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.661459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.661460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.661509 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:32.661657 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:32.661657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:32.661658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:32.661770 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:32.661867 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.661946 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.662003 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:32.663700 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:32.665526 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:32.671696 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.667921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36333 (local address 127.0.0.1:47420)
0619 20:49:32.668390 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.668457 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.668502 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.668758 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.668759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.668761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.668812 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.668814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.669740 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.669741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.670571 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.670574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.671487 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.671487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.671543 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.671554 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.671608 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.671644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260619 20:49:32.671754 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.668073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.98.3:36333)
0619 20:49:32.668492 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.668498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.668508 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:32.668633 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.668635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.668682 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.668763 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.668866 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.669678 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.670658 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.671406 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.671483 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.671549 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.671620 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.671654 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":81,"threads_started":1}
I20260619 20:49:32.672456 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 (DEFAULT_TABLE table=test-table [id=362500d790494d6e9939d12ac14b7d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:32.672787 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.667917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37103 (local address 127.0.0.1:32994)
0619 20:49:32.668069 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.668108 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.668174 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.668400 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.668403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.668405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.668467 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.668468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.669423 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.669424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.670684 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.670687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.672564 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.672564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.672625 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.672630 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.672668 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.672710 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":49,"threads_started":1}
I20260619 20:49:32.672823 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.673046 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.668065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.0.98.1:37103)
0619 20:49:32.668215 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.668220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.668232 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:49:32.668282 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.668284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.668300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.668373 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.668552 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.669291 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.671952 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.672469 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.672552 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.672624 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.672927 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.672976 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260619 20:49:32.673523 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.667921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37077 (local address 127.0.0.1:41762)
0619 20:49:32.668255 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.668265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.668316 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.669568 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.669569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.669571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.669609 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.669610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.670792 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.670793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.671842 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.671845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.673253 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.673254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.673359 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.673363 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.673422 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.673460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":86,"threads_started":1}
I20260619 20:49:32.673583 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.668248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.0.98.2:37077)
0619 20:49:32.669420 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.669424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.669433 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:32.669468 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.669470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.669484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.669559 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.669681 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.670719 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.672109 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.673063 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.673131 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.673219 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.673449 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.673489 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":106,"threads_started":1}
I20260619 20:49:32.673683 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 (DEFAULT_TABLE table=test-table [id=362500d790494d6e9939d12ac14b7d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:32.673980 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.674201 1770 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 (DEFAULT_TABLE table=test-table [id=362500d790494d6e9939d12ac14b7d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:32.674503 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.676551 2010 tablet_bootstrap.cc:492] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Bootstrap starting.
I20260619 20:49:32.676923 2011 tablet_bootstrap.cc:492] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Bootstrap starting.
I20260619 20:49:32.677166 2010 tablet_bootstrap.cc:654] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.677491 2010 log.cc:824] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:32.677573 2011 tablet_bootstrap.cc:654] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.677883 2011 log.cc:824] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:32.678190 2010 tablet_bootstrap.cc:492] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: No bootstrap required, opened a new log
I20260619 20:49:32.678260 2010 ts_tablet_manager.cc:1397] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:32.678278 2014 tablet_bootstrap.cc:492] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Bootstrap starting.
I20260619 20:49:32.678725 2011 tablet_bootstrap.cc:492] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: No bootstrap required, opened a new log
I20260619 20:49:32.678795 2011 ts_tablet_manager.cc:1397] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:32.678867 2014 tablet_bootstrap.cc:654] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.679162 2014 log.cc:824] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:32.679873 2014 tablet_bootstrap.cc:492] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: No bootstrap required, opened a new log
I20260619 20:49:32.679936 2014 ts_tablet_manager.cc:1397] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:32.680179 2010 raft_consensus.cc:348] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.680320 2010 raft_consensus.cc:374] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.680357 2010 raft_consensus.cc:729] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cca9446d5b496f9582c11dc7316e81, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.680464 2010 consensus_queue.cc:260] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.680757 1987 heartbeater.cc:503] Master 127.0.98.62:46487 was elected leader, sending a full tablet report...
I20260619 20:49:32.680748 2011 raft_consensus.cc:348] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.680907 2011 raft_consensus.cc:374] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.680946 2011 raft_consensus.cc:729] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe3924cd0b148a3b1e3e461e5a01464, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.681051 2011 consensus_queue.cc:260] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.681146 2010 ts_tablet_manager.cc:1428] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:49:32.681324 2011 ts_tablet_manager.cc:1428] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:49:32.681439 1707 heartbeater.cc:503] Master 127.0.98.62:46487 was elected leader, sending a full tablet report...
I20260619 20:49:32.681983 2014 raft_consensus.cc:348] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.682127 2014 raft_consensus.cc:374] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.682197 2014 raft_consensus.cc:729] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8759920e944186b4acc4b47dd89698, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.682301 2014 consensus_queue.cc:260] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.682547 2014 ts_tablet_manager.cc:1428] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:49:32.682591 1847 heartbeater.cc:503] Master 127.0.98.62:46487 was elected leader, sending a full tablet report...
W20260619 20:49:32.714095 1848 tablet.cc:2369] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:49:32.788983 1708 tablet.cc:2369] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:32.813763 2018 raft_consensus.cc:482] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:32.813877 2018 raft_consensus.cc:504] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.814194 2018 leader_election.cc:288] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103), 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333)
I20260619 20:49:32.818262 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.814478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49727 (local address 127.0.98.1:37103)
0619 20:49:32.814535 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.814537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.814686 (+ 149us) server_negotiation.cc:407] Connection header received
0619 20:49:32.814759 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.814760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.814774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.814875 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.815017 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.815725 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.817250 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.817858 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.817946 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.817958 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.818147 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.818188 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:49:32.818262 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.814494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58117 (local address 127.0.98.3:36333)
0619 20:49:32.814545 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.814547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.814558 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:49:32.814606 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.814607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.814619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.814719 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.814910 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.815693 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.817250 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.817859 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.817946 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.817958 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.818147 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.818188 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:49:32.818269 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.814434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37103 (local address 127.0.98.2:49727)
0619 20:49:32.814609 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.814617 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.814671 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.814881 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.814882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.814884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.814951 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.814952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.815796 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.815797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.817143 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.817146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.817957 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.817958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.818049 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.818054 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.818117 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.818165 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":65,"thread_start_us":104,"threads_started":1}
I20260619 20:49:32.818274 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.814405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36333 (local address 127.0.98.2:58117)
0619 20:49:32.814465 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.814477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.814518 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.814730 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.814731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.814733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.814810 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.814814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.815789 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.815790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.817143 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.817146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.817957 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.817958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.818049 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.818053 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.818117 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.818165 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":81}
I20260619 20:49:32.818631 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4ef518c7ea499f8f4c40c56c9b9ea8" candidate_uuid: "3e8759920e944186b4acc4b47dd89698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cca9446d5b496f9582c11dc7316e81" is_pre_election: true
I20260619 20:49:32.818629 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4ef518c7ea499f8f4c40c56c9b9ea8" candidate_uuid: "3e8759920e944186b4acc4b47dd89698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" is_pre_election: true
I20260619 20:49:32.818794 1935 raft_consensus.cc:2454] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e8759920e944186b4acc4b47dd89698 in term 0.
I20260619 20:49:32.818794 1661 raft_consensus.cc:2454] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e8759920e944186b4acc4b47dd89698 in term 0.
I20260619 20:49:32.819044 1732 leader_election.cc:302] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8759920e944186b4acc4b47dd89698, 99cca9446d5b496f9582c11dc7316e81; no voters:
I20260619 20:49:32.819204 2018 raft_consensus.cc:2781] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:32.819267 2018 raft_consensus.cc:482] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:32.819298 2018 raft_consensus.cc:3037] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.820335 2018 raft_consensus.cc:504] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.820487 2018 leader_election.cc:288] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103), 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333)
I20260619 20:49:32.820633 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4ef518c7ea499f8f4c40c56c9b9ea8" candidate_uuid: "3e8759920e944186b4acc4b47dd89698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe3924cd0b148a3b1e3e461e5a01464"
I20260619 20:49:32.820695 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4ef518c7ea499f8f4c40c56c9b9ea8" candidate_uuid: "3e8759920e944186b4acc4b47dd89698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cca9446d5b496f9582c11dc7316e81"
I20260619 20:49:32.820736 1661 raft_consensus.cc:3037] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.820760 1935 raft_consensus.cc:3037] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.821614 1661 raft_consensus.cc:2454] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8759920e944186b4acc4b47dd89698 in term 1.
I20260619 20:49:32.821614 1935 raft_consensus.cc:2454] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8759920e944186b4acc4b47dd89698 in term 1.
I20260619 20:49:32.821808 1734 leader_election.cc:302] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8759920e944186b4acc4b47dd89698, 4fe3924cd0b148a3b1e3e461e5a01464; no voters:
I20260619 20:49:32.821902 2018 raft_consensus.cc:2781] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:32.822058 2018 raft_consensus.cc:686] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 1 LEADER]: Becoming Leader. State: Replica: 3e8759920e944186b4acc4b47dd89698, State: Running, Role: LEADER
I20260619 20:49:32.822170 2018 consensus_queue.cc:237] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.822999 1503 catalog_manager.cc:5495] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e8759920e944186b4acc4b47dd89698 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3e8759920e944186b4acc4b47dd89698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260619 20:49:32.852794 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.850617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.0.98.2:37077)
0619 20:49:32.850683 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.850689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.850699 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:32.850755 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.850756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.850777 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.850836 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:32.850939 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.851403 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.852137 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.852320 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.852374 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.852523 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:32.852615 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:32.852675 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.852717 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.852735 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:49:32.854697 1935 raft_consensus.cc:1261] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Refusing update from remote peer 3e8759920e944186b4acc4b47dd89698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:32.854707 1661 raft_consensus.cc:1261] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Refusing update from remote peer 3e8759920e944186b4acc4b47dd89698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:32.855058 2020 consensus_queue.cc:1035] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:32.855250 2018 consensus_queue.cc:1035] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:32.857334 2023 mvcc.cc:205] Tried to move back new op lower bound from 7298671300010188800 to 7298671299880325120. Current Snapshot: MvccSnapshot[applied={T|T < 7298671300010188800}]
I20260619 20:49:32.857594 2024 mvcc.cc:205] Tried to move back new op lower bound from 7298671300010188800 to 7298671299880325120. Current Snapshot: MvccSnapshot[applied={T|T < 7298671300010188800}]
I20260619 20:49:32.857766 2025 mvcc.cc:205] Tried to move back new op lower bound from 7298671300010188800 to 7298671299880325120. Current Snapshot: MvccSnapshot[applied={T|T < 7298671300010188800}]
I20260619 20:49:32.860773 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:32.860869 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:32.861600 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: Sending DeleteTablet for 3 replicas of tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8
I20260619 20:49:32.861901 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32994
I20260619 20:49:32.861948 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47420
I20260619 20:49:32.862149 2033 tablet_replica.cc:330] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: stopping tablet replica
I20260619 20:49:32.862190 1770 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41762
I20260619 20:49:32.862242 2033 raft_consensus.cc:2229] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:32.862195 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 20:49:32.862327 2033 raft_consensus.cc:2258] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:49:32.862334 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:32.862710 2035 tablet_replica.cc:330] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: stopping tablet replica
I20260619 20:49:32.862751 2033 ts_tablet_manager.cc:1905] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:32.862798 2035 raft_consensus.cc:2229] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:32.862879 2035 raft_consensus.cc:2258] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:32.863008 2034 tablet_replica.cc:330] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: stopping tablet replica
I20260619 20:49:32.863737 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8354cc645e4b098613b832192fdbc9 (DEFAULT_TABLE table=test-table [id=b6d6c01520204f698b5e889018747f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:32.863837 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8354cc645e4b098613b832192fdbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.863852 1770 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8354cc645e4b098613b832192fdbc9 (DEFAULT_TABLE table=test-table [id=b6d6c01520204f698b5e889018747f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:32.863967 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8354cc645e4b098613b832192fdbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.863981 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8354cc645e4b098613b832192fdbc9 (DEFAULT_TABLE table=test-table [id=b6d6c01520204f698b5e889018747f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:32.864068 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8354cc645e4b098613b832192fdbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:32.865150 2011 tablet_bootstrap.cc:492] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Bootstrap starting.
I20260619 20:49:32.865322 2010 tablet_bootstrap.cc:492] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Bootstrap starting.
I20260619 20:49:32.865357 2014 tablet_bootstrap.cc:492] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Bootstrap starting.
I20260619 20:49:32.865576 2011 tablet_bootstrap.cc:654] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.865747 2014 tablet_bootstrap.cc:654] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.865756 2010 tablet_bootstrap.cc:654] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:32.866371 2034 raft_consensus.cc:2229] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:32.866490 2034 raft_consensus.cc:2258] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:32.866825 2033 ts_tablet_manager.cc:1918] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:32.866897 2033 log.cc:1196] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/2d4ef518c7ea499f8f4c40c56c9b9ea8
I20260619 20:49:32.866886 2035 ts_tablet_manager.cc:1905] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:32.867188 2033 ts_tablet_manager.cc:1939] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 4fe3924cd0b148a3b1e3e461e5a01464: Deleting consensus metadata
I20260619 20:49:32.867677 1490 catalog_manager.cc:4841] TS 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103): tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 (table test-table [id=362500d790494d6e9939d12ac14b7d84]) successfully deleted
I20260619 20:49:32.867865 2011 tablet_bootstrap.cc:492] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: No bootstrap required, opened a new log
I20260619 20:49:32.867918 2011 ts_tablet_manager.cc:1397] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:49:32.868244 2011 raft_consensus.cc:348] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.868322 2011 raft_consensus.cc:374] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.868352 2011 raft_consensus.cc:729] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe3924cd0b148a3b1e3e461e5a01464, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.868391 2011 consensus_queue.cc:260] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.868489 2011 ts_tablet_manager.cc:1428] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:32.868779 2035 ts_tablet_manager.cc:1918] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:32.868845 2035 log.cc:1196] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/2d4ef518c7ea499f8f4c40c56c9b9ea8
I20260619 20:49:32.869094 2035 ts_tablet_manager.cc:1939] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 3e8759920e944186b4acc4b47dd89698: Deleting consensus metadata
I20260619 20:49:32.869565 1491 catalog_manager.cc:4841] TS 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077): tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 (table test-table [id=362500d790494d6e9939d12ac14b7d84]) successfully deleted
I20260619 20:49:32.869661 2034 ts_tablet_manager.cc:1905] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:32.870908 2034 ts_tablet_manager.cc:1918] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:32.870980 2034 log.cc:1196] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/2d4ef518c7ea499f8f4c40c56c9b9ea8
I20260619 20:49:32.871232 2034 ts_tablet_manager.cc:1939] T 2d4ef518c7ea499f8f4c40c56c9b9ea8 P 99cca9446d5b496f9582c11dc7316e81: Deleting consensus metadata
I20260619 20:49:32.871460 2014 tablet_bootstrap.cc:492] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: No bootstrap required, opened a new log
I20260619 20:49:32.871524 2014 ts_tablet_manager.cc:1397] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260619 20:49:32.871608 1488 catalog_manager.cc:4841] TS 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333): tablet 2d4ef518c7ea499f8f4c40c56c9b9ea8 (table test-table [id=362500d790494d6e9939d12ac14b7d84]) successfully deleted
I20260619 20:49:32.871665 2014 raft_consensus.cc:348] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.871747 2014 raft_consensus.cc:374] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.871778 2014 raft_consensus.cc:729] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8759920e944186b4acc4b47dd89698, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.871824 2014 consensus_queue.cc:260] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.871929 2014 ts_tablet_manager.cc:1428] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:32.877646 2010 tablet_bootstrap.cc:492] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: No bootstrap required, opened a new log
I20260619 20:49:32.877709 2010 ts_tablet_manager.cc:1397] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260619 20:49:32.877875 2010 raft_consensus.cc:348] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.877928 2010 raft_consensus.cc:374] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:32.877941 2010 raft_consensus.cc:729] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cca9446d5b496f9582c11dc7316e81, State: Initialized, Role: FOLLOWER
I20260619 20:49:32.877996 2010 consensus_queue.cc:260] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.878084 2010 ts_tablet_manager.cc:1428] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:32.959915 2017 raft_consensus.cc:482] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:32.960024 2017 raft_consensus.cc:504] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.960340 2017 leader_election.cc:288] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077), 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333)
I20260619 20:49:32.963917 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.960533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36333 (local address 127.0.98.1:34009)
0619 20:49:32.960768 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.960777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.960821 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.960973 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.960974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.960977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.961065 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.961067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.962047 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.962048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.962937 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.962939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.963630 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.963631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.963711 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.963716 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.963780 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.963818 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260619 20:49:32.963914 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.960746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34009 (local address 127.0.98.3:36333)
0619 20:49:32.960785 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.960786 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.960818 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:49:32.960879 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.960879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.960888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.960954 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.961145 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.961939 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.963040 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.963527 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.963599 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.963609 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.963786 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.963822 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:32.964229 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8354cc645e4b098613b832192fdbc9" candidate_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cca9446d5b496f9582c11dc7316e81" is_pre_election: true
I20260619 20:49:32.964310 1935 raft_consensus.cc:2454] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe3924cd0b148a3b1e3e461e5a01464 in term 0.
I20260619 20:49:32.964327 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.960664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42927 (local address 127.0.98.2:37077)
0619 20:49:32.960707 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:32.960709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:32.960789 (+ 80us) server_negotiation.cc:407] Connection header received
0619 20:49:32.960919 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:32.960922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:32.960954 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:32.960998 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:32.961187 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.961934 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.963407 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.963964 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.964042 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.964054 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:32.964222 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 20:49:32.964262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:49:32.964531 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:32.960533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37077 (local address 127.0.98.1:42927)
0619 20:49:32.960705 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:32.960716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:32.960820 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:32.961023 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:32.961025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:32.961027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:32.961099 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.961100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.962016 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.962017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.963320 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:32.963323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:32.964041 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:32.964042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:32.964133 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:32.964137 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:32.964423 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 20:49:32.964458 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":29,"thread_start_us":68,"threads_started":1}
I20260619 20:49:32.964571 1592 leader_election.cc:302] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe3924cd0b148a3b1e3e461e5a01464, 99cca9446d5b496f9582c11dc7316e81; no voters:
I20260619 20:49:32.964794 2017 raft_consensus.cc:2781] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:32.964848 2017 raft_consensus.cc:482] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:32.964870 2017 raft_consensus.cc:3037] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.964906 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8354cc645e4b098613b832192fdbc9" candidate_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8759920e944186b4acc4b47dd89698" is_pre_election: true
I20260619 20:49:32.965037 1801 raft_consensus.cc:2454] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe3924cd0b148a3b1e3e461e5a01464 in term 0.
I20260619 20:49:32.965719 2017 raft_consensus.cc:504] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.965886 2017 leader_election.cc:288] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077), 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333)
I20260619 20:49:32.966019 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8354cc645e4b098613b832192fdbc9" candidate_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8759920e944186b4acc4b47dd89698"
I20260619 20:49:32.966092 1801 raft_consensus.cc:3037] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.966086 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8354cc645e4b098613b832192fdbc9" candidate_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cca9446d5b496f9582c11dc7316e81"
I20260619 20:49:32.966145 1935 raft_consensus.cc:3037] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:32.966889 1801 raft_consensus.cc:2454] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe3924cd0b148a3b1e3e461e5a01464 in term 1.
I20260619 20:49:32.966890 1935 raft_consensus.cc:2454] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe3924cd0b148a3b1e3e461e5a01464 in term 1.
I20260619 20:49:32.967053 1592 leader_election.cc:302] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe3924cd0b148a3b1e3e461e5a01464, 99cca9446d5b496f9582c11dc7316e81; no voters:
I20260619 20:49:32.967141 2017 raft_consensus.cc:2781] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:32.967307 2017 raft_consensus.cc:686] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe3924cd0b148a3b1e3e461e5a01464, State: Running, Role: LEADER
I20260619 20:49:32.967379 2017 consensus_queue.cc:237] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:32.968084 1504 catalog_manager.cc:5495] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
Jun 19 20:49:33 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:33.012115 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.005856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46487 (local address 127.0.0.1:41170)
0619 20:49:33.006009 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:33.006019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:33.006080 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:33.006203 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:33.006204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:33.006499 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:33.006543 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.006545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.007147 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.007148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.007660 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.007661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.007962 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.007962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.008000 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.008527 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:33.008532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:33.009853 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:33.011251 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:33.011257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:33.011259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:33.011435 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:33.011595 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:33.011596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:33.011596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:33.011647 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:33.011839 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:33.011844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:33.011881 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:49:33.011931 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:49:33.012005 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260619 20:49:33.012125 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.005924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.98.62:46487)
0619 20:49:33.006024 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.006026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.006079 (+ 53us) server_negotiation.cc:407] Connection header received
0619 20:49:33.006134 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.006135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.006143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.006185 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:33.006613 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.007082 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.007722 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.007902 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.007951 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.009945 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:33.009947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:33.009947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:33.009953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:33.011173 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:33.011495 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:33.011496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:33.011496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:33.011541 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:33.011696 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:33.011696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:33.011697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:33.011786 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:33.011835 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.011959 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.012046 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":72}
I20260619 20:49:33.014117 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41170:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:49:33.014360 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:33.015818 1641 tablet_service.cc:1467] Processing CreateTablet for tablet fce528fc10074550a0ef44efc145c234 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a269b90b3b204c079adc2587754eb8fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:49:33.015811 1770 tablet_service.cc:1467] Processing CreateTablet for tablet fce528fc10074550a0ef44efc145c234 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a269b90b3b204c079adc2587754eb8fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:49:33.015846 1921 tablet_service.cc:1467] Processing CreateTablet for tablet fce528fc10074550a0ef44efc145c234 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a269b90b3b204c079adc2587754eb8fa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:49:33.015976 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce528fc10074550a0ef44efc145c234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:33.015978 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce528fc10074550a0ef44efc145c234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:33.015980 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce528fc10074550a0ef44efc145c234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:33.017387 2011 tablet_bootstrap.cc:492] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: Bootstrap starting.
I20260619 20:49:33.017422 2014 tablet_bootstrap.cc:492] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: Bootstrap starting.
I20260619 20:49:33.017428 2010 tablet_bootstrap.cc:492] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: Bootstrap starting.
I20260619 20:49:33.017771 2010 tablet_bootstrap.cc:654] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:33.017764 2014 tablet_bootstrap.cc:654] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:33.017791 2011 tablet_bootstrap.cc:654] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:33.018458 2010 tablet_bootstrap.cc:492] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: No bootstrap required, opened a new log
I20260619 20:49:33.018515 2010 ts_tablet_manager.cc:1397] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:33.018676 2010 raft_consensus.cc:348] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.018769 2010 raft_consensus.cc:374] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:33.018790 2014 tablet_bootstrap.cc:492] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: No bootstrap required, opened a new log
I20260619 20:49:33.018805 2010 raft_consensus.cc:729] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cca9446d5b496f9582c11dc7316e81, State: Initialized, Role: FOLLOWER
I20260619 20:49:33.018841 2014 ts_tablet_manager.cc:1397] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:33.018851 2010 consensus_queue.cc:260] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.018898 2011 tablet_bootstrap.cc:492] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: No bootstrap required, opened a new log
I20260619 20:49:33.018949 2011 ts_tablet_manager.cc:1397] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 20:49:33.018949 2010 ts_tablet_manager.cc:1428] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:33.018949 2014 raft_consensus.cc:348] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.019024 2014 raft_consensus.cc:374] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:33.019050 2014 raft_consensus.cc:729] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8759920e944186b4acc4b47dd89698, State: Initialized, Role: FOLLOWER
I20260619 20:49:33.019083 2011 raft_consensus.cc:348] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.019102 2014 consensus_queue.cc:260] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.019070 2016 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.019155 2011 raft_consensus.cc:374] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:33.019158 2016 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:33.019186 2011 raft_consensus.cc:729] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe3924cd0b148a3b1e3e461e5a01464, State: Initialized, Role: FOLLOWER
I20260619 20:49:33.019196 2014 ts_tablet_manager.cc:1428] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:33.019237 2011 consensus_queue.cc:260] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.019228 2020 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.019327 2011 ts_tablet_manager.cc:1428] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:33.019307 2020 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:33.019389 2017 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.019500 2017 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 20:49:33.116324 1988 tablet.cc:2369] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:33.279707 2016 raft_consensus.cc:482] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:33.279808 2016 raft_consensus.cc:504] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.280310 2016 leader_election.cc:288] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103), 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077)
I20260619 20:49:33.283329 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.280558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41301 (local address 127.0.98.1:37103)
0619 20:49:33.280603 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.280606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.280737 (+ 131us) server_negotiation.cc:407] Connection header received
0619 20:49:33.280811 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.280812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.280821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.280898 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:33.281062 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.281558 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.282582 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.282994 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.283048 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.283057 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.283223 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.283267 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:49:33.283341 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.280494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37103 (local address 127.0.98.3:41301)
0619 20:49:33.280630 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:33.280641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:33.280752 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:33.280901 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:33.280902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:33.280904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:33.280990 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.280992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.281628 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.281630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.282437 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.282439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.283064 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.283065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.283133 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.283137 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:33.283214 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:49:33.283254 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":20,"thread_start_us":64,"threads_started":1}
I20260619 20:49:33.283413 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.280588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40031 (local address 127.0.98.2:37077)
0619 20:49:33.280632 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.280634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.280735 (+ 101us) server_negotiation.cc:407] Connection header received
0619 20:49:33.280807 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.280808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.280821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.280902 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:33.281084 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.281623 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.282544 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.283038 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.283105 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.283114 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.283307 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.283343 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:49:33.283445 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.280514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37077 (local address 127.0.98.3:40031)
0619 20:49:33.280633 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:33.280643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:33.280745 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:33.280905 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:33.280906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:33.280908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:33.280997 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.280999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.281705 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.281705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.282439 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.282441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.283144 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.283145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.283221 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.283227 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:49:33.283301 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:49:33.283341 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260619 20:49:33.283658 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce528fc10074550a0ef44efc145c234" candidate_uuid: "99cca9446d5b496f9582c11dc7316e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" is_pre_election: true
I20260619 20:49:33.283751 1661 raft_consensus.cc:2454] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99cca9446d5b496f9582c11dc7316e81 in term 0.
I20260619 20:49:33.283871 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce528fc10074550a0ef44efc145c234" candidate_uuid: "99cca9446d5b496f9582c11dc7316e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8759920e944186b4acc4b47dd89698" is_pre_election: true
I20260619 20:49:33.283936 1874 leader_election.cc:302] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe3924cd0b148a3b1e3e461e5a01464, 99cca9446d5b496f9582c11dc7316e81; no voters:
I20260619 20:49:33.283955 1801 raft_consensus.cc:2454] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99cca9446d5b496f9582c11dc7316e81 in term 0.
I20260619 20:49:33.284057 2016 raft_consensus.cc:2781] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:33.284125 2016 raft_consensus.cc:482] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:33.284152 2016 raft_consensus.cc:3037] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:33.284864 2016 raft_consensus.cc:504] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.285013 2016 leader_election.cc:288] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103), 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077)
I20260619 20:49:33.285151 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce528fc10074550a0ef44efc145c234" candidate_uuid: "99cca9446d5b496f9582c11dc7316e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8759920e944186b4acc4b47dd89698"
I20260619 20:49:33.285231 1801 raft_consensus.cc:3037] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:33.285208 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce528fc10074550a0ef44efc145c234" candidate_uuid: "99cca9446d5b496f9582c11dc7316e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe3924cd0b148a3b1e3e461e5a01464"
I20260619 20:49:33.285274 1661 raft_consensus.cc:3037] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:33.285877 1801 raft_consensus.cc:2454] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99cca9446d5b496f9582c11dc7316e81 in term 1.
I20260619 20:49:33.285878 1661 raft_consensus.cc:2454] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99cca9446d5b496f9582c11dc7316e81 in term 1.
I20260619 20:49:33.286087 1875 leader_election.cc:302] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8759920e944186b4acc4b47dd89698, 99cca9446d5b496f9582c11dc7316e81; no voters:
I20260619 20:49:33.286206 2016 raft_consensus.cc:2781] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:33.286334 2016 raft_consensus.cc:686] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [term 1 LEADER]: Becoming Leader. State: Replica: 99cca9446d5b496f9582c11dc7316e81, State: Running, Role: LEADER
I20260619 20:49:33.286402 2016 consensus_queue.cc:237] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } }
I20260619 20:49:33.286702 2054 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: TxnStatusTablet state changed. Reason: New leader 99cca9446d5b496f9582c11dc7316e81. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.286796 2054 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:49:33.286980 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:49:33.287117 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:49:33.287087 1503 catalog_manager.cc:5495] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } health_report { overall_health: HEALTHY } } }
I20260619 20:49:33.302963 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.300521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.0.98.3:36333)
0619 20:49:33.300563 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.300565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.300574 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:33.300667 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.300668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.300683 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.300785 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:33.300919 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.301441 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.302192 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.302398 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.302463 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.302650 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.302747 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.302822 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.302864 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.302881 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:49:33.302977 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.300456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36333 (local address 127.0.0.1:47432)
0619 20:49:33.300525 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:33.300536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:33.300584 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:33.300791 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:33.300792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:33.300793 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:33.300843 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.300845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.301517 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.301519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.302100 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.302102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.302462 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.302462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.302523 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.302538 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.302809 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.302817 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:49:33.302861 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:49:33.302906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:49:33.304080 1661 raft_consensus.cc:1261] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Refusing update from remote peer 99cca9446d5b496f9582c11dc7316e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:33.304141 1801 raft_consensus.cc:1261] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Refusing update from remote peer 99cca9446d5b496f9582c11dc7316e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:33.304353 2016 consensus_queue.cc:1035] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:33.304459 2054 consensus_queue.cc:1035] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:33.304941 2017 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: TxnStatusTablet state changed. Reason: New leader 99cca9446d5b496f9582c11dc7316e81. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.305034 2017 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:33.305042 2020 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: TxnStatusTablet state changed. Reason: New leader 99cca9446d5b496f9582c11dc7316e81. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.305145 2020 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:33.305804 2054 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.305874 2054 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:49:33.306022 2020 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.306093 2020 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 3e8759920e944186b4acc4b47dd89698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:33.306581 2016 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.306665 2016 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 99cca9446d5b496f9582c11dc7316e81: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:49:33.306998 2017 tablet_replica.cc:439] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99cca9446d5b496f9582c11dc7316e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe3924cd0b148a3b1e3e461e5a01464" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37103 } } peers { permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 } } peers { permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 } } }
I20260619 20:49:33.307086 2017 tablet_replica.cc:442] T fce528fc10074550a0ef44efc145c234 P 4fe3924cd0b148a3b1e3e461e5a01464: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:33.310312 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.307528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.0.98.1:37103)
0619 20:49:33.307598 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.307600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.307852 (+ 252us) server_negotiation.cc:407] Connection header received
0619 20:49:33.307904 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.307905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.307921 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.307997 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:33.308070 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.308790 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.309521 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.309812 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.309886 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.309983 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.310089 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.310189 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.310227 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.310246 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 20:49:33.315552 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.312617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47271 (local address 127.0.98.3:36333)
0619 20:49:33.312668 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.312669 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.312788 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:49:33.312873 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.312874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.312883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.312950 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:33.313114 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.313667 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.314833 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.315072 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.315138 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.315313 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.315349 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.315409 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.315467 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.315484 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:33.315753 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.312558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36333 (local address 127.0.98.1:47271)
0619 20:49:33.312723 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:33.312732 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:33.312803 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:33.312953 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:33.312954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:33.312955 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:33.313043 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.313044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.313761 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.313762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.314738 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.314740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.315139 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.315140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.315210 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.315222 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.315410 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.315412 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:33.315464 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:49:33.315503 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260619 20:49:33.317682 1935 raft_consensus.cc:1261] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Refusing update from remote peer 4fe3924cd0b148a3b1e3e461e5a01464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:33.317696 1801 raft_consensus.cc:1261] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Refusing update from remote peer 4fe3924cd0b148a3b1e3e461e5a01464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:33.317921 2041 consensus_queue.cc:1035] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99cca9446d5b496f9582c11dc7316e81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:33.318140 2041 consensus_queue.cc:1035] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8759920e944186b4acc4b47dd89698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:33.326576 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.323962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37183 (local address 127.0.98.1:37103)
0619 20:49:33.324001 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.324003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.324128 (+ 125us) server_negotiation.cc:407] Connection header received
0619 20:49:33.324198 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.324199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.324207 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.324247 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:33.324400 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.324861 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.325784 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.326065 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.326155 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.326344 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.326378 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.326432 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.326490 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.326508 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:33.326614 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.323924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37103 (local address 127.0.98.3:37183)
0619 20:49:33.324073 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:33.324085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:33.324142 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:33.324257 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:33.324258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:33.324274 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:33.324326 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.324327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.324948 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.324950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.325688 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:33.325691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.326165 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.326166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:33.326252 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.326264 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:33.326433 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:33.326435 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:33.326486 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:49:33.326524 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260619 20:49:33.332849 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:33.332949 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:33.333537 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd2f86d25d1b406d815339f44dceb387: Sending DeleteTablet for 3 replicas of tablet 0f8354cc645e4b098613b832192fdbc9
I20260619 20:49:33.333782 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8354cc645e4b098613b832192fdbc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47420
I20260619 20:49:33.333793 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8354cc645e4b098613b832192fdbc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32994
I20260619 20:49:33.333892 1770 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8354cc645e4b098613b832192fdbc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41762
I20260619 20:49:33.333976 2034 tablet_replica.cc:330] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: stopping tablet replica
I20260619 20:49:33.334028 2034 raft_consensus.cc:2229] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:33.334059 2034 raft_consensus.cc:2258] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:33.333891 2033 tablet_replica.cc:330] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: stopping tablet replica
I20260619 20:49:33.334113 2035 tablet_replica.cc:330] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: stopping tablet replica
I20260619 20:49:33.334167 2035 raft_consensus.cc:2229] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:33.334195 2035 raft_consensus.cc:2258] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:33.334184 2033 raft_consensus.cc:2229] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:33.334281 2033 raft_consensus.cc:2258] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:33.334532 2033 ts_tablet_manager.cc:1905] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:33.334807 2035 ts_tablet_manager.cc:1905] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:33.336325 2035 ts_tablet_manager.cc:1918] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:33.336372 2035 log.cc:1196] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/0f8354cc645e4b098613b832192fdbc9
I20260619 20:49:33.336577 2035 ts_tablet_manager.cc:1939] T 0f8354cc645e4b098613b832192fdbc9 P 3e8759920e944186b4acc4b47dd89698: Deleting consensus metadata
I20260619 20:49:33.336869 1491 catalog_manager.cc:4841] TS 3e8759920e944186b4acc4b47dd89698 (127.0.98.2:37077): tablet 0f8354cc645e4b098613b832192fdbc9 (table test-table [id=b6d6c01520204f698b5e889018747f14]) successfully deleted
I20260619 20:49:33.337071 2034 ts_tablet_manager.cc:1905] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:33.337423 2033 ts_tablet_manager.cc:1918] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:33.337481 2033 log.cc:1196] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/0f8354cc645e4b098613b832192fdbc9
I20260619 20:49:33.337703 2033 ts_tablet_manager.cc:1939] T 0f8354cc645e4b098613b832192fdbc9 P 4fe3924cd0b148a3b1e3e461e5a01464: Deleting consensus metadata
I20260619 20:49:33.338953 2034 ts_tablet_manager.cc:1918] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:33.339012 2034 log.cc:1196] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/0f8354cc645e4b098613b832192fdbc9
I20260619 20:49:33.339226 2034 ts_tablet_manager.cc:1939] T 0f8354cc645e4b098613b832192fdbc9 P 99cca9446d5b496f9582c11dc7316e81: Deleting consensus metadata
I20260619 20:49:33.339514 1488 catalog_manager.cc:4841] TS 99cca9446d5b496f9582c11dc7316e81 (127.0.98.3:36333): tablet 0f8354cc645e4b098613b832192fdbc9 (table test-table [id=b6d6c01520204f698b5e889018747f14]) successfully deleted
I20260619 20:49:33.339669 1490 catalog_manager.cc:4841] TS 4fe3924cd0b148a3b1e3e461e5a01464 (127.0.98.1:37103): tablet 0f8354cc645e4b098613b832192fdbc9 (table test-table [id=b6d6c01520204f698b5e889018747f14]) successfully deleted
Jun 19 20:49:33 dist-test-slave-0smx krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:33.346719 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.336806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.0.98.1:37103)
0619 20:49:33.336975 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.336977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.336983 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:49:33.336997 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.336998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.337005 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.337064 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:33.337880 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.338554 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.339596 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.339896 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.339939 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.343475 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:33.343481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:33.343482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:33.343488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:33.345457 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:33.345843 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:33.345846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:33.345846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:33.345918 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:33.346148 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:33.346149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:33.346149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:33.346301 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:33.346443 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.346506 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.346581 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
W20260619 20:49:33.346928 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33012
I20260619 20:49:33.354974 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:33.347866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.98.62:46487)
0619 20:49:33.347906 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:33.347908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:33.348099 (+ 191us) server_negotiation.cc:407] Connection header received
0619 20:49:33.348133 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:33.348134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:33.348145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:33.348197 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:33.348807 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.349594 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.350592 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:33.350895 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:33.350965 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:33.352243 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:33.352244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:33.352244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:33.352251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:33.353820 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:33.354187 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:33.354188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:33.354189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:33.354247 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:33.354450 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:33.354450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:33.354451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:33.354583 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:33.354678 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:33.354818 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 20:49:33.354888 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:49:33.355507 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41178
I20260619 20:49:33.356204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1576
I20260619 20:49:33.364359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1716
I20260619 20:49:33.370553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1856
I20260619 20:49:33.374917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 1472
2026-06-19T20:49:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2563 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:33 dist-test-slave-0smx krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:33 dist-test-slave-0smx krb5kdc[2078](info): set up 2 sockets
Jun 19 20:49:33 dist-test-slave-0smx krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260619 20:49:34.420974 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:34.532424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:34.611340 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:34.611537 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:34.611558 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:34.612661 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:34.612702 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:49:34.612715 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:34.612727 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:34.612739 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:34.612751 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:34.614239 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:34.614485 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:34.614701 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:34.619728 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:34.619733 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:34.619719 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:34.619976 2091 server_base.cc:1034] running on GCE node
I20260619 20:49:34.620221 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:34.620545 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:34.621708 2091 hybrid_clock.cc:648] HybridClock initialized: now 1781902174621684 us; error 38 us; skew 500 ppm
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:34.624043 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:34.624475 2091 webserver.cc:460] Webserver started at http://127.0.98.62:35933/ using document root <none> and password file <none>
I20260619 20:49:34.624704 2091 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:34.624770 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:34.624918 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:34.625967 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "44adc2809d744d90a345a6bdf3f52f28"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "32e6f26139afe4c4d8d54f15f9a81cb3"
server_key_iv: "9d74629a1f9a1aa75c17b3ecb180e754"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.626296 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "44adc2809d744d90a345a6bdf3f52f28"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "32e6f26139afe4c4d8d54f15f9a81cb3"
server_key_iv: "9d74629a1f9a1aa75c17b3ecb180e754"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.627488 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:34.628190 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:34.628356 2091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:34.628450 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "44adc2809d744d90a345a6bdf3f52f28"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "32e6f26139afe4c4d8d54f15f9a81cb3"
server_key_iv: "9d74629a1f9a1aa75c17b3ecb180e754"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.628525 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:34.647303 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:34.649641 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:34.649796 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:34.654388 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45805
I20260619 20:49:34.654434 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45805 every 8 connection(s)
I20260619 20:49:34.654755 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:34.655332 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:34.656752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 2091
I20260619 20:49:34.656849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:34.657007 392 external_mini_cluster.cc:1428] Setting key 18ccd84b1385ceeef2ff653fd3823699
I20260619 20:49:34.657968 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Bootstrap starting.
I20260619 20:49:34.658638 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:34.658922 2169 log.cc:824] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:34.659631 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: No bootstrap required, opened a new log
I20260619 20:49:34.661934 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44adc2809d744d90a345a6bdf3f52f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260619 20:49:34.662122 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:34.662163 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44adc2809d744d90a345a6bdf3f52f28, State: Initialized, Role: FOLLOWER
I20260619 20:49:34.662284 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44adc2809d744d90a345a6bdf3f52f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260619 20:49:34.662354 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:34.662377 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:34.662443 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:34.662921 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44adc2809d744d90a345a6bdf3f52f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260619 20:49:34.662999 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44adc2809d744d90a345a6bdf3f52f28; no voters:
I20260619 20:49:34.663177 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:34.663206 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:34.663316 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [term 1 LEADER]: Becoming Leader. State: Replica: 44adc2809d744d90a345a6bdf3f52f28, State: Running, Role: LEADER
I20260619 20:49:34.663390 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44adc2809d744d90a345a6bdf3f52f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260619 20:49:34.663417 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:34.663743 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44adc2809d744d90a345a6bdf3f52f28. Latest consensus state: current_term: 1 leader_uuid: "44adc2809d744d90a345a6bdf3f52f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44adc2809d744d90a345a6bdf3f52f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } } }
I20260619 20:49:34.663810 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:34.663765 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44adc2809d744d90a345a6bdf3f52f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44adc2809d744d90a345a6bdf3f52f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } } }
I20260619 20:49:34.663944 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28 [sys.catalog]: This master's current role is: LEADER
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260619 20:49:34.665321 2189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:49:34.665367 2189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:49:34.665426 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:34.665850 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:34.667426 2179 catalog_manager.cc:1347] Generated new cluster ID: b59ef83839694712a0a7cb8c7a963f82
I20260619 20:49:34.667470 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:34.668264 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.657545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.98.62:45805)
0619 20:49:34.657741 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:34.657750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:34.657781 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:49:34.658317 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:34.658327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:34.658400 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:34.658563 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:34.659215 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.660553 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.661315 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.661939 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.662051 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.664610 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:34.664615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:34.664615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:34.664629 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:34.666664 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.667028 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.667030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.667031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.667133 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.667347 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.667348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.667348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.667643 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:34.667778 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:34.667944 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 20:49:34.668071 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260619 20:49:34.678618 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:34.679116 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:34.683725 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Generated new TSK 0
I20260619 20:49:34.683885 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:34.708492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45805
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:34.787032 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:34.787230 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:34.787254 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:34.788404 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:34.788451 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:34.788471 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:49:34.788519 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:49:34.788551 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:34.790060 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:34.790295 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:34.790516 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:34.795378 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:34.795397 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:34.795428 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:34.795655 2195 server_base.cc:1034] running on GCE node
I20260619 20:49:34.795871 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:34.796139 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:34.797361 2195 hybrid_clock.cc:648] HybridClock initialized: now 1781902174797345 us; error 46 us; skew 500 ppm
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:34.799764 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:34.800235 2195 webserver.cc:460] Webserver started at http://127.0.98.1:40337/ using document root <none> and password file <none>
I20260619 20:49:34.800459 2195 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:34.800504 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:34.800644 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:34.801730 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ff2e74f36294ac895490dd33eb58582"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "84c91fbb69318967a8fe77c54c448711"
server_key_iv: "80fa7601b51947833a11f92e43612f5c"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.802073 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ff2e74f36294ac895490dd33eb58582"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "84c91fbb69318967a8fe77c54c448711"
server_key_iv: "80fa7601b51947833a11f92e43612f5c"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.803295 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:34.803987 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:34.804152 2195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:34.804241 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "5ff2e74f36294ac895490dd33eb58582"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "84c91fbb69318967a8fe77c54c448711"
server_key_iv: "80fa7601b51947833a11f92e43612f5c"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.804314 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:34.815670 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:34.817423 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:34.817571 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:34.818085 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:34.818140 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:34.818179 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:34.818198 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:34.825049 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35295
I20260619 20:49:34.825380 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35295 every 8 connection(s)
I20260619 20:49:34.825462 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:34.827378 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.818396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33453 (local address 127.0.98.62:45805)
0619 20:49:34.818444 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:34.818447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:34.818705 (+ 258us) server_negotiation.cc:407] Connection header received
0619 20:49:34.819135 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:34.819137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:34.819151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:34.819235 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:34.819770 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.820583 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.821870 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.822100 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.822187 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.824873 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:34.824874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:34.824875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:34.824883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:34.826329 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.826662 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.826663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.826663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.826709 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.826870 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.826871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.826871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.826992 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:34.827040 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:34.827256 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 20:49:34.827318 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:34.827541 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.818391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.98.1:33453)
0619 20:49:34.818573 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:34.818612 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:34.819042 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:34.819281 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:34.819283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:34.819493 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:34.819676 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.819680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.820701 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.820702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.821775 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.821778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.822208 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.822209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.822339 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.822932 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:34.822950 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:34.824770 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:34.826394 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.826398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.826400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.826601 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.826747 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.826747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.826747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.826825 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.827044 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:34.827048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:34.827157 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 20:49:34.827234 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:49:34.827338 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260619 20:49:34.828110 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45805
I20260619 20:49:34.828191 2326 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:34.828359 2326 heartbeater.cc:511] Master 127.0.98.62:45805 requested a full tablet report, sending...
I20260619 20:49:34.828863 2123 ts_manager.cc:194] Registered new tserver with Master: 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295)
I20260619 20:49:34.829761 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33453
I20260619 20:49:34.832659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 2195
I20260619 20:49:34.832767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:34.832957 392 external_mini_cluster.cc:1428] Setting key aee33591431ba34d82d45def666ead3b
I20260619 20:49:34.834491 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.829097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.98.1:58647)
0619 20:49:34.829212 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:34.829222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:34.829276 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:34.829456 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:34.829457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:34.829598 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:34.829647 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.829649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.830455 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.830456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.830978 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.830980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.831294 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.831294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.831350 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.831783 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:34.831787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:34.832034 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:34.833662 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.833663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.833664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.833828 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.834015 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.834015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.834016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.834058 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.834280 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:34.834280 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:34.834309 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 20:49:34.834353 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:49:34.834418 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260619 20:49:34.834580 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.829149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58647 (local address 127.0.98.62:45805)
0619 20:49:34.829204 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:34.829206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:34.829267 (+ 61us) server_negotiation.cc:407] Connection header received
0619 20:49:34.829363 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:34.829364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:34.829372 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:34.829452 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:34.829712 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.830335 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.831050 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.831242 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.831294 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.832108 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:34.832110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:34.832110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:34.832115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:34.833575 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.833897 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.833897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.833898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.833954 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.834101 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.834102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.834102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.834227 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:34.834292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:34.834377 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:49:34.834455 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:34.856663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45805
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:34.935648 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:34.935838 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:34.935863 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:34.936896 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:34.936929 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:34.936949 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:49:34.936960 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:49:34.936976 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:34.938410 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:34.938576 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:34.938732 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:34.943446 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:34.943439 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:34.943544 2335 server_base.cc:1034] running on GCE node
W20260619 20:49:34.943434 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:34.943812 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:34.944042 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:34.945204 2335 hybrid_clock.cc:648] HybridClock initialized: now 1781902174945189 us; error 30 us; skew 500 ppm
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:34.947525 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:34.947974 2335 webserver.cc:460] Webserver started at http://127.0.98.2:37389/ using document root <none> and password file <none>
I20260619 20:49:34.948145 2335 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:34.948184 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:34.948264 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:34.949196 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "4acf21755b2f42908e8b769e14a6f143"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "0f4547549b58d71a7d63a1782b1ef259"
server_key_iv: "a6d694db841679c4f9927f6947cac58e"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.949556 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4acf21755b2f42908e8b769e14a6f143"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "0f4547549b58d71a7d63a1782b1ef259"
server_key_iv: "a6d694db841679c4f9927f6947cac58e"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.950764 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:34.951404 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:34.951587 2335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:34.951673 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "4acf21755b2f42908e8b769e14a6f143"
format_stamp: "Formatted at 2026-06-19 20:49:34 on dist-test-slave-0smx"
server_key: "0f4547549b58d71a7d63a1782b1ef259"
server_key_iv: "a6d694db841679c4f9927f6947cac58e"
server_key_version: "encryptionkey@0"
I20260619 20:49:34.951725 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:34.960846 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:34.963104 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:34.963238 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:34.963738 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:34.963783 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:34.963821 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:34.963846 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:34.970125 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36931
I20260619 20:49:34.970206 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36931 every 8 connection(s)
I20260619 20:49:34.970484 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:49:34 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:34.970781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 2335
I20260619 20:49:34.970862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:34.971047 392 external_mini_cluster.cc:1428] Setting key 256f6d7eb172fd3057498b520134d873
I20260619 20:49:34.974036 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.964113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33219 (local address 127.0.98.62:45805)
0619 20:49:34.964207 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:34.964209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:34.964511 (+ 302us) server_negotiation.cc:407] Connection header received
0619 20:49:34.964949 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:34.964951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:34.964964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:34.965049 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:34.965943 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.966741 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.968274 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.968603 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.968689 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.971180 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:34.971182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:34.971182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:34.971190 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:34.972613 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.973057 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.973059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.973059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.973118 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.973404 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.973405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.973406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.973522 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:34.973595 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:34.973873 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 20:49:34.973949 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:49:34.974218 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.964209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.98.2:33219)
0619 20:49:34.964381 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:34.964418 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:34.964859 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:34.965178 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:34.965183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:34.965561 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:34.965843 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.965850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.966838 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.966839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.968178 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.968181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.968689 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.968690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.968784 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.969349 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:34.969359 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:34.971063 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:34.972716 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.972722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.972724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.972966 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.973192 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.973193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.973193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.973348 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.973597 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:34.973603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:34.973732 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 20:49:34.973838 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 20:49:34.973985 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260619 20:49:34.974674 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45805
I20260619 20:49:34.974754 2466 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:34.974925 2466 heartbeater.cc:511] Master 127.0.98.62:45805 requested a full tablet report, sending...
I20260619 20:49:34.975298 2122 ts_manager.cc:194] Registered new tserver with Master: 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931)
I20260619 20:49:34.975800 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33219
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 20:49:34.983319 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.975700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54693 (local address 127.0.98.62:45805)
0619 20:49:34.975918 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:34.975920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:34.977484 (+ 1564us) server_negotiation.cc:407] Connection header received
0619 20:49:34.977546 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:34.977547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:34.977555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:34.977602 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:34.977951 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.978390 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.979452 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.979631 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.979697 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.980585 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:34.980587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:34.980587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:34.980592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:34.982177 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.982571 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.982571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.982571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.982616 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:34.982808 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:34.982808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:34.982809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:34.982939 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:34.983017 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:34.983173 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:49:34.983236 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260619 20:49:34.983332 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:34.975642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.98.2:54693)
0619 20:49:34.977407 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:34.977419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:34.977483 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:34.977608 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:34.977609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:34.977814 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:34.977877 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.977879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.978486 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.978487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.979362 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:34.979366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:34.979706 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:34.979707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:34.979764 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:34.980251 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:34.980256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:34.980504 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:34.982258 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.982259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.982260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.982495 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.982679 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:34.982680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:34.982681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:34.982749 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:34.983013 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:34.983014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:34.983056 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 20:49:34.983137 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:49:34.983233 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:34.995596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45805
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:35.075850 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:35.076067 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:35.076105 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:35.077210 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:35.077275 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:35.077328 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:49:35.077363 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:49:35.077399 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:35.078795 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:35.079061 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:35.079282 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:35.084144 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:35.084270 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:35.084151 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:35.085052 2475 server_base.cc:1034] running on GCE node
I20260619 20:49:35.085225 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:35.085477 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:35.086642 2475 hybrid_clock.cc:648] HybridClock initialized: now 1781902175086619 us; error 42 us; skew 500 ppm
Jun 19 20:49:35 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:35.088910 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:35.089257 2475 webserver.cc:460] Webserver started at http://127.0.98.3:38063/ using document root <none> and password file <none>
I20260619 20:49:35.089504 2475 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:35.089550 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:35.089666 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:35.090611 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "2584614f312f44e68913f7f5e2714b3d"
format_stamp: "Formatted at 2026-06-19 20:49:35 on dist-test-slave-0smx"
server_key: "bf70ade9295b9bfe213140c9eb7564aa"
server_key_iv: "ee90978991be932062f5985c82a4216a"
server_key_version: "encryptionkey@0"
I20260619 20:49:35.090901 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2584614f312f44e68913f7f5e2714b3d"
format_stamp: "Formatted at 2026-06-19 20:49:35 on dist-test-slave-0smx"
server_key: "bf70ade9295b9bfe213140c9eb7564aa"
server_key_iv: "ee90978991be932062f5985c82a4216a"
server_key_version: "encryptionkey@0"
I20260619 20:49:35.092077 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:35.092715 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.092898 2475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:35.092981 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "2584614f312f44e68913f7f5e2714b3d"
format_stamp: "Formatted at 2026-06-19 20:49:35 on dist-test-slave-0smx"
server_key: "bf70ade9295b9bfe213140c9eb7564aa"
server_key_iv: "ee90978991be932062f5985c82a4216a"
server_key_version: "encryptionkey@0"
I20260619 20:49:35.093027 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:35.106213 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:35.108464 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:35.108577 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:35.108992 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:35.109028 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.109076 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:35.109102 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.115660 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39341
I20260619 20:49:35.115743 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39341 every 8 connection(s)
I20260619 20:49:35.116036 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:49:35 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:35.118939 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.109466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51999 (local address 127.0.98.62:45805)
0619 20:49:35.109537 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.109539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.109763 (+ 224us) server_negotiation.cc:407] Connection header received
0619 20:49:35.110275 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.110277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.110291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.110370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:35.111194 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.112053 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.113616 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.113935 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.114019 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.116644 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:35.116645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:35.116645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:35.116651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:35.117865 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.118219 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.118220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.118220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.118268 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.118475 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.118475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.118475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.118571 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:35.118621 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.118816 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.118878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:49:35.119109 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.109440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.98.3:51999)
0619 20:49:35.109629 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.109666 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.110176 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.110454 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.110458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.110802 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:35.111086 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.111092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.112158 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.112159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.113511 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.113514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.114022 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.114022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.114125 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.114703 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:35.114712 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:35.116540 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:35.117938 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:35.117943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:35.117945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:35.118155 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:35.118324 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:35.118324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:35.118325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:35.118416 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:35.118625 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:35.118629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:35.118730 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.118807 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.118916 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20260619 20:49:35.119604 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45805
I20260619 20:49:35.119689 2606 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:35.119861 2606 heartbeater.cc:511] Master 127.0.98.62:45805 requested a full tablet report, sending...
I20260619 20:49:35.120132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 2475
I20260619 20:49:35.120247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:35.120249 2122 ts_manager.cc:194] Registered new tserver with Master: 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341)
I20260619 20:49:35.120438 392 external_mini_cluster.cc:1428] Setting key 955a87c30371b1d40b1b6ae3c15f4e80
I20260619 20:49:35.120728 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51999
I20260619 20:49:35.121277 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:49:35.126466 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.120833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38053 (local address 127.0.98.62:45805)
0619 20:49:35.120870 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.120872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.120935 (+ 63us) server_negotiation.cc:407] Connection header received
0619 20:49:35.121010 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.121011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.121020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.121075 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:35.121442 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.121998 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.122688 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.122892 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.122984 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.123953 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:35.123954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:35.123954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:35.123960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:35.125374 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.125741 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.125741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.125742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.125786 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.125990 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.125991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.125991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.126077 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:35.126158 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.126311 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.126380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:49:35.126480 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.120661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.98.3:38053)
0619 20:49:35.120843 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.120889 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.120957 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.121081 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.121081 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.121272 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:35.121350 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.121352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.122122 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.122123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.122604 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.122605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.122985 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.122986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.123064 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.123461 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:35.123466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:35.123868 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:35.125447 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:35.125448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:35.125449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:35.125666 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:35.125853 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:35.125854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:35.125854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:35.125923 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:35.126152 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:35.126153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:35.126196 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.126285 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.126383 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
Jun 19 20:49:35 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:49:35 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:35.145105 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.138025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.0.98.62:45805)
0619 20:49:35.138087 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.138089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.138099 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:35.138171 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.138172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.138185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.138253 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:35.138877 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.139367 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.140240 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.140462 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.140556 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.142883 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:35.142885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:35.142885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:35.142893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:35.143998 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.144329 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.144330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.144331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.144386 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.144595 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.144595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.144596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.144711 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:35.144829 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.144912 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.145000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:49:35.146471 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:35.147946 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:35.154289 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.150194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35295 (local address 127.0.0.1:56796)
0619 20:49:35.150347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.150357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.150438 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.150931 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.150932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.150934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.150971 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.150972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.152231 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.152232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.153151 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.153154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.154010 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.154011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.154086 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.154090 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.154121 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.154180 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260619 20:49:35.154320 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.150060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.0.98.3:39341)
0619 20:49:35.150701 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.150706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.150716 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:35.150752 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.150754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.150769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.151008 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.151062 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.151708 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.153389 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.153842 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.153961 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.154015 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.154157 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.154191 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":76,"threads_started":1}
I20260619 20:49:35.154294 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.149992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39341 (local address 127.0.0.1:47454)
0619 20:49:35.150156 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.150196 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.150264 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.150867 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.150868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.150870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.150934 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.150937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.151795 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.151796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.153079 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.153080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.153965 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.153966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.154058 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.154070 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.154140 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.154187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":7,"thread_start_us":63,"threads_started":1}
I20260619 20:49:35.154412 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.150100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.98.2:36931)
0619 20:49:35.150401 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.150406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.150416 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:35.150453 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.150454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.150469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.150570 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.150774 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.151923 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.153164 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.153870 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.153936 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.153993 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.154270 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.154306 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":65,"threads_started":1}
I20260619 20:49:35.154589 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.149998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.0.1:39592)
0619 20:49:35.150239 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.150250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.150290 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.150615 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.150618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.150620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.150678 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.150680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.152004 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.152005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.153077 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.153080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.154035 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.154035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.154096 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.154099 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.154145 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.154180 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":25,"thread_start_us":71,"threads_started":1}
I20260619 20:49:35.155217 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.150458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.0.98.1:35295)
0619 20:49:35.150761 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.150766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.150776 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:35.150811 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.150814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.150830 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.151037 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.151095 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.152126 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.153240 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.153931 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.155025 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.155085 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.155108 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.155145 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260619 20:49:35.155402 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 5b97492046974eb08736c1dd722f8fbb (DEFAULT_TABLE table=test-table [id=b36ac431411e4634968f5cab8f829999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:35.155411 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 5b97492046974eb08736c1dd722f8fbb (DEFAULT_TABLE table=test-table [id=b36ac431411e4634968f5cab8f829999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:35.155759 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b97492046974eb08736c1dd722f8fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.155758 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b97492046974eb08736c1dd722f8fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.155786 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 5b97492046974eb08736c1dd722f8fbb (DEFAULT_TABLE table=test-table [id=b36ac431411e4634968f5cab8f829999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:35.156122 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b97492046974eb08736c1dd722f8fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.158818 2626 tablet_bootstrap.cc:492] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Bootstrap starting.
I20260619 20:49:35.159246 2627 tablet_bootstrap.cc:492] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Bootstrap starting.
I20260619 20:49:35.159314 2628 tablet_bootstrap.cc:492] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Bootstrap starting.
I20260619 20:49:35.159667 2626 tablet_bootstrap.cc:654] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.160032 2628 tablet_bootstrap.cc:654] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.160008 2626 log.cc:824] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:35.160089 2627 tablet_bootstrap.cc:654] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.160338 2628 log.cc:824] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:35.160426 2627 log.cc:824] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:35.160828 2626 tablet_bootstrap.cc:492] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: No bootstrap required, opened a new log
I20260619 20:49:35.160909 2626 ts_tablet_manager.cc:1397] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:35.161118 2628 tablet_bootstrap.cc:492] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: No bootstrap required, opened a new log
I20260619 20:49:35.161126 2627 tablet_bootstrap.cc:492] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: No bootstrap required, opened a new log
I20260619 20:49:35.161185 2628 ts_tablet_manager.cc:1397] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:35.161206 2627 ts_tablet_manager.cc:1397] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:35.162743 2627 raft_consensus.cc:348] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.162896 2627 raft_consensus.cc:374] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.162938 2627 raft_consensus.cc:729] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.162969 2626 raft_consensus.cc:348] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.163058 2627 consensus_queue.cc:260] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.163157 2626 raft_consensus.cc:374] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.163092 2628 raft_consensus.cc:348] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.163199 2626 raft_consensus.cc:729] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff2e74f36294ac895490dd33eb58582, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.163223 2628 raft_consensus.cc:374] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.163254 2628 raft_consensus.cc:729] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.163329 2627 ts_tablet_manager.cc:1428] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:49:35.163342 2628 consensus_queue.cc:260] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.163451 2606 heartbeater.cc:503] Master 127.0.98.62:45805 was elected leader, sending a full tablet report...
I20260619 20:49:35.163641 2466 heartbeater.cc:503] Master 127.0.98.62:45805 was elected leader, sending a full tablet report...
I20260619 20:49:35.163328 2626 consensus_queue.cc:260] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.163626 2628 ts_tablet_manager.cc:1428] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:49:35.163901 2626 ts_tablet_manager.cc:1428] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:49:35.164104 2326 heartbeater.cc:503] Master 127.0.98.62:45805 was elected leader, sending a full tablet report...
I20260619 20:49:35.194886 2633 raft_consensus.cc:482] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:35.194970 2633 raft_consensus.cc:504] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.195319 2633 leader_election.cc:288] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341), 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295)
I20260619 20:49:35.198334 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.195530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35785 (local address 127.0.98.3:39341)
0619 20:49:35.195572 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.195574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.195585 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:49:35.195685 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.195686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.195697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.195752 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.195909 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.196397 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.197530 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.198031 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.198089 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.198097 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.198260 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.198287 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:35.198370 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.195485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39341 (local address 127.0.98.2:35785)
0619 20:49:35.195528 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.195539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.195577 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.195756 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.195758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.195759 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.195841 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.195842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.196508 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.196509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.197454 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.197456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.198105 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.198106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.198187 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.198191 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.198250 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.198288 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 20:49:35.198700 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b97492046974eb08736c1dd722f8fbb" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2584614f312f44e68913f7f5e2714b3d" is_pre_election: true
I20260619 20:49:35.198807 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.195646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42349 (local address 127.0.98.1:35295)
0619 20:49:35.195678 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.195680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.195749 (+ 69us) server_negotiation.cc:407] Connection header received
0619 20:49:35.195815 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.195816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.195827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.195878 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.195997 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.196667 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.197872 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.198325 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.198387 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.198396 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.198715 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.198744 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:49:35.198817 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.195581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35295 (local address 127.0.98.2:42349)
0619 20:49:35.195703 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.195710 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.195752 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.195883 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.195884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.195885 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.195947 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.195948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.196746 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.196747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.197791 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.197793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.198561 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.198562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.198636 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.198640 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.198695 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.198737 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260619 20:49:35.198838 2560 raft_consensus.cc:2454] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 0.
I20260619 20:49:35.199090 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b97492046974eb08736c1dd722f8fbb" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582" is_pre_election: true
I20260619 20:49:35.199213 2353 leader_election.cc:302] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2584614f312f44e68913f7f5e2714b3d, 4acf21755b2f42908e8b769e14a6f143; no voters:
I20260619 20:49:35.199239 2280 raft_consensus.cc:2454] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 0.
I20260619 20:49:35.199353 2633 raft_consensus.cc:2781] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:35.199400 2633 raft_consensus.cc:482] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:35.199427 2633 raft_consensus.cc:3037] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.200080 2633 raft_consensus.cc:504] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.200204 2633 leader_election.cc:288] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 election: Requested vote from peers 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341), 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295)
I20260619 20:49:35.200344 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b97492046974eb08736c1dd722f8fbb" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2584614f312f44e68913f7f5e2714b3d"
I20260619 20:49:35.200409 2560 raft_consensus.cc:3037] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.200408 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b97492046974eb08736c1dd722f8fbb" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582"
I20260619 20:49:35.200479 2280 raft_consensus.cc:3037] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.201242 2560 raft_consensus.cc:2454] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 1.
I20260619 20:49:35.201247 2280 raft_consensus.cc:2454] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 1.
I20260619 20:49:35.201431 2353 leader_election.cc:302] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2584614f312f44e68913f7f5e2714b3d, 4acf21755b2f42908e8b769e14a6f143; no voters:
I20260619 20:49:35.201521 2633 raft_consensus.cc:2781] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:35.201670 2633 raft_consensus.cc:686] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 1 LEADER]: Becoming Leader. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Running, Role: LEADER
I20260619 20:49:35.201789 2633 consensus_queue.cc:237] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } }
I20260619 20:49:35.202643 2122 catalog_manager.cc:5495] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } health_report { overall_health: UNKNOWN } } }
W20260619 20:49:35.221282 2467 tablet.cc:2369] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:49:35 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:35.256538 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.250110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45805 (local address 127.0.0.1:38052)
0619 20:49:35.250294 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.250305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.250377 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.250512 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.250513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.250816 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:35.250870 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.250872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.251494 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.251495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.252048 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.252050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.252353 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.252354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.252394 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.252866 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:35.252870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:35.254221 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:35.255560 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:35.255567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:35.255568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:35.255795 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:35.255982 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:35.255983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:35.255983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:35.256044 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:35.256249 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:35.256255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:35.256293 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.256351 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.256443 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260619 20:49:35.256582 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.250175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.98.62:45805)
0619 20:49:35.250281 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.250283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.250377 (+ 94us) server_negotiation.cc:407] Connection header received
0619 20:49:35.250433 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.250434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.250442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.250493 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:35.250944 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.251423 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.252119 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.252287 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.252336 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.254291 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:35.254292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:35.254292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:35.254298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:35.255494 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.255864 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.255865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.255865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.255911 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:35.256100 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:35.256101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:35.256101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:35.256194 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:35.256241 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.256379 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.256484 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":76}
I20260619 20:49:35.258662 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38052:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:49:35.258848 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:35.260437 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 0acf87fdca53441396ae7b96778c52f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc89ff13bbd34efbb1100aa7d8c8d907]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:49:35.260447 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 0acf87fdca53441396ae7b96778c52f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc89ff13bbd34efbb1100aa7d8c8d907]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:49:35.260602 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acf87fdca53441396ae7b96778c52f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.260625 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acf87fdca53441396ae7b96778c52f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.261469 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 0acf87fdca53441396ae7b96778c52f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc89ff13bbd34efbb1100aa7d8c8d907]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:49:35.261621 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acf87fdca53441396ae7b96778c52f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.262006 2626 tablet_bootstrap.cc:492] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: Bootstrap starting.
I20260619 20:49:35.262662 2626 tablet_bootstrap.cc:654] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.262975 2627 tablet_bootstrap.cc:492] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: Bootstrap starting.
I20260619 20:49:35.262979 2628 tablet_bootstrap.cc:492] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: Bootstrap starting.
I20260619 20:49:35.263262 2626 tablet_bootstrap.cc:492] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: No bootstrap required, opened a new log
I20260619 20:49:35.263304 2626 ts_tablet_manager.cc:1397] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:35.263373 2628 tablet_bootstrap.cc:654] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.263511 2627 tablet_bootstrap.cc:654] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.263470 2626 raft_consensus.cc:348] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.263557 2626 raft_consensus.cc:374] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.263600 2626 raft_consensus.cc:729] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff2e74f36294ac895490dd33eb58582, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.263649 2626 consensus_queue.cc:260] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.263754 2626 ts_tablet_manager.cc:1428] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:35.263851 2634 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.263942 2634 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:35.264137 2628 tablet_bootstrap.cc:492] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: No bootstrap required, opened a new log
I20260619 20:49:35.264179 2628 ts_tablet_manager.cc:1397] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:35.264326 2628 raft_consensus.cc:348] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.264392 2628 raft_consensus.cc:374] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.264416 2628 raft_consensus.cc:729] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.264479 2628 consensus_queue.cc:260] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.264596 2628 ts_tablet_manager.cc:1428] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:35.264716 2633 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.264783 2633 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:35.264786 2627 tablet_bootstrap.cc:492] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: No bootstrap required, opened a new log
I20260619 20:49:35.264833 2627 ts_tablet_manager.cc:1397] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:35.265007 2627 raft_consensus.cc:348] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.265105 2627 raft_consensus.cc:374] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.265131 2627 raft_consensus.cc:729] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.265187 2627 consensus_queue.cc:260] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.265296 2627 ts_tablet_manager.cc:1428] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.265393 2632 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.265511 2632 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 20:49:35.326193 2327 tablet.cc:2369] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:49:35.366693 2607 tablet.cc:2369] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:35.476585 2633 raft_consensus.cc:482] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:35.476673 2633 raft_consensus.cc:504] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.476784 2633 leader_election.cc:288] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295), 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341)
I20260619 20:49:35.477028 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acf87fdca53441396ae7b96778c52f9" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582" is_pre_election: true
I20260619 20:49:35.477121 2280 raft_consensus.cc:2454] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 0.
I20260619 20:49:35.477105 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acf87fdca53441396ae7b96778c52f9" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2584614f312f44e68913f7f5e2714b3d" is_pre_election: true
I20260619 20:49:35.477223 2560 raft_consensus.cc:2454] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 0.
I20260619 20:49:35.477280 2353 leader_election.cc:302] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acf21755b2f42908e8b769e14a6f143, 5ff2e74f36294ac895490dd33eb58582; no voters:
I20260619 20:49:35.477448 2633 raft_consensus.cc:2781] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:35.477502 2633 raft_consensus.cc:482] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:35.477517 2633 raft_consensus.cc:3037] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.478176 2633 raft_consensus.cc:504] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.478267 2633 leader_election.cc:288] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295), 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341)
I20260619 20:49:35.478453 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acf87fdca53441396ae7b96778c52f9" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582"
I20260619 20:49:35.478536 2280 raft_consensus.cc:3037] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.478513 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acf87fdca53441396ae7b96778c52f9" candidate_uuid: "4acf21755b2f42908e8b769e14a6f143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2584614f312f44e68913f7f5e2714b3d"
I20260619 20:49:35.478603 2560 raft_consensus.cc:3037] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.479166 2560 raft_consensus.cc:2454] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 1.
I20260619 20:49:35.479166 2280 raft_consensus.cc:2454] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acf21755b2f42908e8b769e14a6f143 in term 1.
I20260619 20:49:35.479352 2353 leader_election.cc:302] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acf21755b2f42908e8b769e14a6f143, 5ff2e74f36294ac895490dd33eb58582; no voters:
I20260619 20:49:35.479489 2633 raft_consensus.cc:2781] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:35.479568 2633 raft_consensus.cc:686] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [term 1 LEADER]: Becoming Leader. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Running, Role: LEADER
I20260619 20:49:35.479614 2633 consensus_queue.cc:237] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.479898 2636 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: TxnStatusTablet state changed. Reason: New leader 4acf21755b2f42908e8b769e14a6f143. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.479990 2636 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:49:35.480213 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:49:35.480255 2122 catalog_manager.cc:5495] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } health_report { overall_health: UNKNOWN } } }
I20260619 20:49:35.480361 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:49:35.489799 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.487249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.98.2:36931)
0619 20:49:35.487295 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.487297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.487356 (+ 59us) server_negotiation.cc:407] Connection header received
0619 20:49:35.487450 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.487451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.487467 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.487560 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:35.487721 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.488202 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.488953 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.489171 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.489255 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.489470 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.489567 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.489655 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.489697 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.489731 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:49:35.489807 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.487185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.0.1:39600)
0619 20:49:35.487311 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.487322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.487376 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.487565 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.487566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.487567 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:35.487624 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.487626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.488282 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.488283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.488870 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.488871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.489257 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.489258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.489345 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.489360 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.489633 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.489641 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.489691 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.489739 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 20:49:35.491379 2560 raft_consensus.cc:1261] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Refusing update from remote peer 4acf21755b2f42908e8b769e14a6f143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:35.491417 2280 raft_consensus.cc:1261] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Refusing update from remote peer 4acf21755b2f42908e8b769e14a6f143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:35.491645 2633 consensus_queue.cc:1035] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:35.491740 2636 consensus_queue.cc:1035] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:35.492830 2632 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: TxnStatusTablet state changed. Reason: New leader 4acf21755b2f42908e8b769e14a6f143. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.492941 2632 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:35.493005 2634 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: TxnStatusTablet state changed. Reason: New leader 4acf21755b2f42908e8b769e14a6f143. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.493072 2634 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:35.493984 2633 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.494062 2633 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:49:35.494133 2651 mvcc.cc:205] Tried to move back new op lower bound from 7298671310810632192 to 7298671310764920832. Current Snapshot: MvccSnapshot[applied={T|T < 7298671310810632192}]
I20260619 20:49:35.494369 2636 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.494434 2636 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 4acf21755b2f42908e8b769e14a6f143: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:49:35.494779 2632 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.494854 2632 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 2584614f312f44e68913f7f5e2714b3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:35.494944 2634 tablet_replica.cc:439] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4acf21755b2f42908e8b769e14a6f143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } } }
I20260619 20:49:35.495012 2634 tablet_replica.cc:442] T 0acf87fdca53441396ae7b96778c52f9 P 5ff2e74f36294ac895490dd33eb58582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:49:35.495323 2652 mvcc.cc:205] Tried to move back new op lower bound from 7298671310810632192 to 7298671310764920832. Current Snapshot: MvccSnapshot[applied={T|T < 7298671310810632192 or (T in {7298671310810632192})}]
I20260619 20:49:35.498463 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.496207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.0.98.2:36931)
0619 20:49:35.496247 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.496249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.496258 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:35.496324 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.496324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.496333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.496384 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:35.496531 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.496964 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.497807 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.498007 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.498089 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.498231 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.498260 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.498324 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.498377 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.498391 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:49:35.502966 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.500616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46309 (local address 127.0.98.2:36931)
0619 20:49:35.500659 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.500660 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.500811 (+ 151us) server_negotiation.cc:407] Connection header received
0619 20:49:35.500869 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.500869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.500878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.500925 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:35.501060 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.501560 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.502303 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.502470 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.502541 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.502703 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.502732 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.502793 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.502848 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.502879 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":22}
I20260619 20:49:35.503021 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.500567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.98.2:46309)
0619 20:49:35.500708 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.500716 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.500813 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.500925 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.500926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.500927 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:35.500983 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.500984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.501638 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.501639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.502212 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.502213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.502539 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.502540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.502593 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.502605 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.502793 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.502795 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.502845 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.502893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260619 20:49:35.504699 2280 raft_consensus.cc:1261] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Refusing update from remote peer 4acf21755b2f42908e8b769e14a6f143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:35.504712 2560 raft_consensus.cc:1261] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Refusing update from remote peer 4acf21755b2f42908e8b769e14a6f143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:35.504909 2633 consensus_queue.cc:1035] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:35.504973 2636 consensus_queue.cc:1035] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:35.525620 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:35.525730 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:35.526494 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Sending DeleteTablet for 3 replicas of tablet 5b97492046974eb08736c1dd722f8fbb
I20260619 20:49:35.526796 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b97492046974eb08736c1dd722f8fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39592
I20260619 20:49:35.526930 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b97492046974eb08736c1dd722f8fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47454
I20260619 20:49:35.527081 2667 tablet_replica.cc:330] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: stopping tablet replica
I20260619 20:49:35.527115 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b97492046974eb08736c1dd722f8fbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56796
I20260619 20:49:35.527180 2667 raft_consensus.cc:2229] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:35.527132 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:35.527251 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:35.527263 2667 raft_consensus.cc:2258] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:35.527292 2668 tablet_replica.cc:330] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: stopping tablet replica
I20260619 20:49:35.527374 2668 raft_consensus.cc:2229] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:35.527419 2668 raft_consensus.cc:2258] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:35.527768 2667 ts_tablet_manager.cc:1905] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:35.527849 2668 ts_tablet_manager.cc:1905] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:35.528683 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc0405627047b6b5b972fdb6218b45 (DEFAULT_TABLE table=test-table [id=0c5c8c08440e4aa0b1bb1810f7c44956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:35.528829 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc0405627047b6b5b972fdb6218b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.529649 2668 ts_tablet_manager.cc:1918] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:35.529718 2668 log.cc:1196] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/5b97492046974eb08736c1dd722f8fbb
I20260619 20:49:35.529982 2668 ts_tablet_manager.cc:1939] T 5b97492046974eb08736c1dd722f8fbb P 2584614f312f44e68913f7f5e2714b3d: Deleting consensus metadata
I20260619 20:49:35.530042 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc0405627047b6b5b972fdb6218b45 (DEFAULT_TABLE table=test-table [id=0c5c8c08440e4aa0b1bb1810f7c44956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:35.530146 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc0405627047b6b5b972fdb6218b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.530267 2628 tablet_bootstrap.cc:492] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Bootstrap starting.
I20260619 20:49:35.530335 2109 catalog_manager.cc:4841] TS 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341): tablet 5b97492046974eb08736c1dd722f8fbb (table test-table [id=b36ac431411e4634968f5cab8f829999]) successfully deleted
I20260619 20:49:35.530613 2628 tablet_bootstrap.cc:654] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.530992 2667 ts_tablet_manager.cc:1918] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:35.531050 2667 log.cc:1196] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/5b97492046974eb08736c1dd722f8fbb
I20260619 20:49:35.531180 2628 tablet_bootstrap.cc:492] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: No bootstrap required, opened a new log
I20260619 20:49:35.531281 2628 ts_tablet_manager.cc:1397] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:35.531454 2667 ts_tablet_manager.cc:1939] T 5b97492046974eb08736c1dd722f8fbb P 4acf21755b2f42908e8b769e14a6f143: Deleting consensus metadata
I20260619 20:49:35.531417 2628 raft_consensus.cc:348] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.531491 2628 raft_consensus.cc:374] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.531513 2628 raft_consensus.cc:729] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.531551 2628 consensus_queue.cc:260] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.531635 2628 ts_tablet_manager.cc:1428] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.531642 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc0405627047b6b5b972fdb6218b45 (DEFAULT_TABLE table=test-table [id=0c5c8c08440e4aa0b1bb1810f7c44956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:35.531731 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc0405627047b6b5b972fdb6218b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:35.532563 2627 tablet_bootstrap.cc:492] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Bootstrap starting.
I20260619 20:49:35.532614 2626 tablet_bootstrap.cc:492] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Bootstrap starting.
I20260619 20:49:35.532953 2627 tablet_bootstrap.cc:654] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.533015 2626 tablet_bootstrap.cc:654] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:35.533469 2107 catalog_manager.cc:4841] TS 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931): tablet 5b97492046974eb08736c1dd722f8fbb (table test-table [id=b36ac431411e4634968f5cab8f829999]) successfully deleted
I20260619 20:49:35.533583 2627 tablet_bootstrap.cc:492] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: No bootstrap required, opened a new log
I20260619 20:49:35.533623 2627 ts_tablet_manager.cc:1397] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:35.533763 2627 raft_consensus.cc:348] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.533819 2627 raft_consensus.cc:374] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.533849 2627 raft_consensus.cc:729] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.533905 2627 consensus_queue.cc:260] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.533990 2627 ts_tablet_manager.cc:1428] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.534632 2626 tablet_bootstrap.cc:492] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: No bootstrap required, opened a new log
I20260619 20:49:35.534706 2626 ts_tablet_manager.cc:1397] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:35.534848 2626 raft_consensus.cc:348] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.534924 2626 raft_consensus.cc:374] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:35.534956 2626 raft_consensus.cc:729] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff2e74f36294ac895490dd33eb58582, State: Initialized, Role: FOLLOWER
I20260619 20:49:35.535002 2626 consensus_queue.cc:260] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.535085 2626 ts_tablet_manager.cc:1428] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:35.535205 2669 tablet_replica.cc:330] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: stopping tablet replica
I20260619 20:49:35.535305 2669 raft_consensus.cc:2229] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:35.535377 2669 raft_consensus.cc:2258] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:35.535770 2669 ts_tablet_manager.cc:1905] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:35.537194 2669 ts_tablet_manager.cc:1918] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:35.537281 2669 log.cc:1196] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/5b97492046974eb08736c1dd722f8fbb
I20260619 20:49:35.537521 2669 ts_tablet_manager.cc:1939] T 5b97492046974eb08736c1dd722f8fbb P 5ff2e74f36294ac895490dd33eb58582: Deleting consensus metadata
I20260619 20:49:35.537957 2109 catalog_manager.cc:4841] TS 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295): tablet 5b97492046974eb08736c1dd722f8fbb (table test-table [id=b36ac431411e4634968f5cab8f829999]) successfully deleted
I20260619 20:49:35.563140 2632 raft_consensus.cc:482] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:35.563217 2632 raft_consensus.cc:504] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.563552 2632 leader_election.cc:288] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295), 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931)
I20260619 20:49:35.566987 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.563694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.98.3:50489)
0619 20:49:35.563781 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.563793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.563839 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.564069 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.564070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.564071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.564153 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.564155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.564816 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.564818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.566034 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.566036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.566733 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.566734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.566813 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.566817 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.566870 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.566907 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 20:49:35.567003 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.563913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50489 (local address 127.0.98.2:36931)
0619 20:49:35.563953 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.563955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.563962 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:35.563987 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.563988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.563998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.564064 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.564230 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.564743 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.566133 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.566647 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.566729 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.566741 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.566902 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.566937 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:35.567366 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0405627047b6b5b972fdb6218b45" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf21755b2f42908e8b769e14a6f143" is_pre_election: true
I20260619 20:49:35.567474 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.563819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36475 (local address 127.0.98.1:35295)
0619 20:49:35.563893 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.563895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.563936 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:49:35.564019 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.564020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.564029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.564086 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:35.564295 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.565023 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.566562 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.567107 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.567175 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.567184 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.567361 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.567410 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:49:35.567502 2417 raft_consensus.cc:2454] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 0.
I20260619 20:49:35.567684 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.563698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35295 (local address 127.0.98.3:36475)
0619 20:49:35.563876 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.563884 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.563964 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.564133 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.564134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.564136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:35.564212 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.564214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.565102 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.565103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.566476 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.566478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.567174 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.567175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.567254 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.567256 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.567536 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.567576 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":11,"thread_start_us":58,"threads_started":1}
I20260619 20:49:35.567747 2491 leader_election.cc:302] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2584614f312f44e68913f7f5e2714b3d, 4acf21755b2f42908e8b769e14a6f143; no voters:
I20260619 20:49:35.567922 2632 raft_consensus.cc:2781] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:35.567972 2632 raft_consensus.cc:482] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:35.567992 2632 raft_consensus.cc:3037] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.568022 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0405627047b6b5b972fdb6218b45" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582" is_pre_election: true
I20260619 20:49:35.568099 2280 raft_consensus.cc:2454] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 0.
I20260619 20:49:35.568722 2632 raft_consensus.cc:504] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.568873 2632 leader_election.cc:288] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 election: Requested vote from peers 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295), 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931)
I20260619 20:49:35.569041 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0405627047b6b5b972fdb6218b45" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf21755b2f42908e8b769e14a6f143"
I20260619 20:49:35.569118 2417 raft_consensus.cc:3037] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.569099 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0405627047b6b5b972fdb6218b45" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582"
I20260619 20:49:35.569175 2280 raft_consensus.cc:3037] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:35.569748 2280 raft_consensus.cc:2454] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 1.
I20260619 20:49:35.569931 2493 leader_election.cc:302] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2584614f312f44e68913f7f5e2714b3d, 5ff2e74f36294ac895490dd33eb58582; no voters:
I20260619 20:49:35.569936 2417 raft_consensus.cc:2454] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 1.
I20260619 20:49:35.570013 2632 raft_consensus.cc:2781] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:35.570147 2632 raft_consensus.cc:686] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 1 LEADER]: Becoming Leader. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Running, Role: LEADER
I20260619 20:49:35.570209 2632 consensus_queue.cc:237] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:35.570798 2121 catalog_manager.cc:5495] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2584614f312f44e68913f7f5e2714b3d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2584614f312f44e68913f7f5e2714b3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } health_report { overall_health: HEALTHY } } }
I20260619 20:49:35.586241 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.583798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.0.98.3:39341)
0619 20:49:35.583837 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.583839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.583878 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:49:35.583910 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.583911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.583929 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.583993 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:35.584087 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.584761 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.585500 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.585758 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.585829 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.585929 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.586046 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.586112 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.586149 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.586168 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:35.590993 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.588490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.98.3:57057)
0619 20:49:35.588571 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.588584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.588626 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.588815 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.588816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.588817 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:35.588861 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.588862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.589695 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.589695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.590210 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.590211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.590641 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.590642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.590687 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.590716 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.590881 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.590882 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.590916 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.590945 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:49:35.591015 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.588531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57057 (local address 127.0.98.2:36931)
0619 20:49:35.588699 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.588701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.588708 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:35.588727 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.588728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.588736 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.588800 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:35.588927 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.589622 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.590285 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.590570 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.590638 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.590788 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.590822 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.590881 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.590918 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.590934 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260619 20:49:35.593466 2417 raft_consensus.cc:1261] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Refusing update from remote peer 2584614f312f44e68913f7f5e2714b3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:35.593725 2280 raft_consensus.cc:1261] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Refusing update from remote peer 2584614f312f44e68913f7f5e2714b3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:35.593732 2632 consensus_queue.cc:1035] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:35.593946 2632 consensus_queue.cc:1035] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:35.594971 2653 mvcc.cc:205] Tried to move back new op lower bound from 7298671311229018112 to 7298671311136219136. Current Snapshot: MvccSnapshot[applied={T|T < 7298671311229018112}]
I20260619 20:49:35.601164 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.598541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47495 (local address 127.0.98.3:39341)
0619 20:49:35.598578 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:35.598580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:35.598588 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:35.598648 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:35.598649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:35.598658 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:35.598720 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:35.598836 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.599361 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.600389 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.600691 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.600763 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.600931 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.600963 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.601022 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:35.601090 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:49:35.601108 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:35.601197 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:35.598499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39341 (local address 127.0.98.2:47495)
0619 20:49:35.598544 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:35.598554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:35.598586 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:35.598724 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:35.598725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:35.598726 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:35.598763 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.598764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.599445 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.599446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.600273 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:35.600276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:35.600798 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:35.600799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:35.600854 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:35.600860 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:35.601038 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:35.601040 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:35.601087 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 20:49:35.601128 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:49:35.601513 2653 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 20:49:37.125598 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:37.125698 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:37.126446 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Sending DeleteTablet for 3 replicas of tablet 7dbc0405627047b6b5b972fdb6218b45
I20260619 20:49:37.126677 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc0405627047b6b5b972fdb6218b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39592
I20260619 20:49:37.126698 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc0405627047b6b5b972fdb6218b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47454
I20260619 20:49:37.126744 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc0405627047b6b5b972fdb6218b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56796
I20260619 20:49:37.126909 2703 tablet_replica.cc:330] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: stopping tablet replica
I20260619 20:49:37.126936 2705 tablet_replica.cc:330] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: stopping tablet replica
I20260619 20:49:37.127018 2705 raft_consensus.cc:2229] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:37.127079 2705 raft_consensus.cc:2258] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:37.129220 2703 raft_consensus.cc:2229] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:37.129334 2703 raft_consensus.cc:2258] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:37.129451 2704 tablet_replica.cc:330] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: stopping tablet replica
I20260619 20:49:37.129536 2704 raft_consensus.cc:2229] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:37.129597 2703 ts_tablet_manager.cc:1905] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:37.129633 2704 raft_consensus.cc:2258] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:37.129709 2705 ts_tablet_manager.cc:1905] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:37.131105 2703 ts_tablet_manager.cc:1918] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:49:37.131181 2703 log.cc:1196] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/7dbc0405627047b6b5b972fdb6218b45
I20260619 20:49:37.131393 2704 ts_tablet_manager.cc:1905] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:37.131496 2703 ts_tablet_manager.cc:1939] T 7dbc0405627047b6b5b972fdb6218b45 P 4acf21755b2f42908e8b769e14a6f143: Deleting consensus metadata
I20260619 20:49:37.131644 2705 ts_tablet_manager.cc:1918] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:49:37.131700 2705 log.cc:1196] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/7dbc0405627047b6b5b972fdb6218b45
I20260619 20:49:37.131814 2107 catalog_manager.cc:4841] TS 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931): tablet 7dbc0405627047b6b5b972fdb6218b45 (table test-table [id=0c5c8c08440e4aa0b1bb1810f7c44956]) successfully deleted
I20260619 20:49:37.132001 2705 ts_tablet_manager.cc:1939] T 7dbc0405627047b6b5b972fdb6218b45 P 5ff2e74f36294ac895490dd33eb58582: Deleting consensus metadata
I20260619 20:49:37.132296 2109 catalog_manager.cc:4841] TS 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295): tablet 7dbc0405627047b6b5b972fdb6218b45 (table test-table [id=0c5c8c08440e4aa0b1bb1810f7c44956]) successfully deleted
I20260619 20:49:37.132937 2704 ts_tablet_manager.cc:1918] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:49:37.132985 2704 log.cc:1196] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/7dbc0405627047b6b5b972fdb6218b45
I20260619 20:49:37.133205 2704 ts_tablet_manager.cc:1939] T 7dbc0405627047b6b5b972fdb6218b45 P 2584614f312f44e68913f7f5e2714b3d: Deleting consensus metadata
I20260619 20:49:37.133523 2109 catalog_manager.cc:4841] TS 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341): tablet 7dbc0405627047b6b5b972fdb6218b45 (table test-table [id=0c5c8c08440e4aa0b1bb1810f7c44956]) successfully deleted
Jun 19 20:49:37 dist-test-slave-0smx krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:49:37 dist-test-slave-0smx krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:37.156594 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.146126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.98.62:45805)
0619 20:49:37.146257 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.146259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.146381 (+ 122us) server_negotiation.cc:407] Connection header received
0619 20:49:37.146411 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.146412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.146423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.146485 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:37.147010 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.147546 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.148262 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.148455 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.148510 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.154076 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:37.154077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:37.154078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:37.154085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:37.155600 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:37.155910 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:37.155911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:37.155911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:37.155961 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:37.156117 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:37.156117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:37.156118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:37.156211 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:37.156280 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.156390 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.156451 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260619 20:49:37.157955 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:37.158082 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:37.159289 2260 tablet_service.cc:1467] Processing CreateTablet for tablet d2102c46ce294915b3e2aa62fce083af (DEFAULT_TABLE table=test-table [id=bc6a4a5e5f7b41c5855f59d44c9957a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:37.159308 2400 tablet_service.cc:1467] Processing CreateTablet for tablet d2102c46ce294915b3e2aa62fce083af (DEFAULT_TABLE table=test-table [id=bc6a4a5e5f7b41c5855f59d44c9957a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:37.159430 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2102c46ce294915b3e2aa62fce083af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:37.159430 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2102c46ce294915b3e2aa62fce083af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:37.160434 2714 tablet_bootstrap.cc:492] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Bootstrap starting.
I20260619 20:49:37.160591 2715 tablet_bootstrap.cc:492] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Bootstrap starting.
I20260619 20:49:37.160588 2540 tablet_service.cc:1467] Processing CreateTablet for tablet d2102c46ce294915b3e2aa62fce083af (DEFAULT_TABLE table=test-table [id=bc6a4a5e5f7b41c5855f59d44c9957a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:37.160694 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2102c46ce294915b3e2aa62fce083af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:37.160847 2714 tablet_bootstrap.cc:654] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:37.161028 2715 tablet_bootstrap.cc:654] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:37.161645 2715 tablet_bootstrap.cc:492] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: No bootstrap required, opened a new log
I20260619 20:49:37.161695 2715 ts_tablet_manager.cc:1397] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:37.161818 2714 tablet_bootstrap.cc:492] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: No bootstrap required, opened a new log
I20260619 20:49:37.161876 2714 ts_tablet_manager.cc:1397] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:37.161880 2715 raft_consensus.cc:348] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.161962 2715 raft_consensus.cc:374] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:37.161981 2715 raft_consensus.cc:729] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff2e74f36294ac895490dd33eb58582, State: Initialized, Role: FOLLOWER
I20260619 20:49:37.162003 2714 raft_consensus.cc:348] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.162066 2714 raft_consensus.cc:374] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:37.162058 2718 tablet_bootstrap.cc:492] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Bootstrap starting.
I20260619 20:49:37.162045 2715 consensus_queue.cc:260] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.162084 2714 raft_consensus.cc:729] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Initialized, Role: FOLLOWER
I20260619 20:49:37.162147 2714 consensus_queue.cc:260] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.162199 2715 ts_tablet_manager.cc:1428] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:37.162240 2714 ts_tablet_manager.cc:1428] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:37.162500 2718 tablet_bootstrap.cc:654] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:37.163146 2718 tablet_bootstrap.cc:492] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: No bootstrap required, opened a new log
I20260619 20:49:37.163197 2718 ts_tablet_manager.cc:1397] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:37.163301 2718 raft_consensus.cc:348] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.163359 2718 raft_consensus.cc:374] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:37.163371 2718 raft_consensus.cc:729] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Initialized, Role: FOLLOWER
I20260619 20:49:37.163452 2718 consensus_queue.cc:260] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.163544 2718 ts_tablet_manager.cc:1428] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:37.337646 2719 raft_consensus.cc:482] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:37.337741 2719 raft_consensus.cc:504] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.338176 2719 leader_election.cc:288] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931), 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341)
I20260619 20:49:37.341480 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.338486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36287 (local address 127.0.98.2:36931)
0619 20:49:37.338615 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.338618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.338685 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:49:37.338764 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.338765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.338776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.338855 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:37.339010 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.339694 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.340708 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.341088 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.341170 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.341183 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.341370 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.341407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260619 20:49:37.341485 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.338431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39719 (local address 127.0.98.3:39341)
0619 20:49:37.338685 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.338687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.338695 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:37.338726 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.338727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.338737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.338825 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:37.338996 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.339682 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.340702 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.341080 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.341172 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.341183 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.341366 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.341413 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":78,"threads_started":1}
I20260619 20:49:37.341508 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.338363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.98.1:36287)
0619 20:49:37.338589 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:37.338600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:37.338692 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:37.338857 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:37.338858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:37.338860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:37.338931 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.338933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.339771 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.339772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.340614 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.340616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.341169 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.341169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.341247 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.341252 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:37.341363 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 20:49:37.341405 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":50,"thread_start_us":145,"threads_started":1}
I20260619 20:49:37.341691 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.338363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39341 (local address 127.0.98.1:39719)
0619 20:49:37.338601 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:37.338612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:37.338670 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:37.338827 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:37.338828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:37.338830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:37.338918 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.338920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.339757 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.339758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.340610 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.340612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.341176 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.341177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.341255 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.341259 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:37.341355 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:49:37.341397 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":45,"thread_start_us":135,"threads_started":1}
I20260619 20:49:37.341925 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2102c46ce294915b3e2aa62fce083af" candidate_uuid: "5ff2e74f36294ac895490dd33eb58582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf21755b2f42908e8b769e14a6f143" is_pre_election: true
I20260619 20:49:37.342027 2417 raft_consensus.cc:2454] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff2e74f36294ac895490dd33eb58582 in term 0.
I20260619 20:49:37.342084 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2102c46ce294915b3e2aa62fce083af" candidate_uuid: "5ff2e74f36294ac895490dd33eb58582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2584614f312f44e68913f7f5e2714b3d" is_pre_election: true
I20260619 20:49:37.342180 2560 raft_consensus.cc:2454] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff2e74f36294ac895490dd33eb58582 in term 0.
I20260619 20:49:37.342182 2211 leader_election.cc:302] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acf21755b2f42908e8b769e14a6f143, 5ff2e74f36294ac895490dd33eb58582; no voters:
I20260619 20:49:37.342303 2719 raft_consensus.cc:2781] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:37.342342 2719 raft_consensus.cc:482] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:37.342374 2719 raft_consensus.cc:3037] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:37.343183 2719 raft_consensus.cc:504] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.343338 2719 leader_election.cc:288] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [CANDIDATE]: Term 1 election: Requested vote from peers 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931), 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341)
I20260619 20:49:37.343461 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2102c46ce294915b3e2aa62fce083af" candidate_uuid: "5ff2e74f36294ac895490dd33eb58582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf21755b2f42908e8b769e14a6f143"
I20260619 20:49:37.343538 2417 raft_consensus.cc:3037] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:37.343503 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2102c46ce294915b3e2aa62fce083af" candidate_uuid: "5ff2e74f36294ac895490dd33eb58582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2584614f312f44e68913f7f5e2714b3d"
I20260619 20:49:37.343571 2560 raft_consensus.cc:3037] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:37.344133 2417 raft_consensus.cc:2454] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff2e74f36294ac895490dd33eb58582 in term 1.
I20260619 20:49:37.344211 2560 raft_consensus.cc:2454] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff2e74f36294ac895490dd33eb58582 in term 1.
I20260619 20:49:37.344300 2211 leader_election.cc:302] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acf21755b2f42908e8b769e14a6f143, 5ff2e74f36294ac895490dd33eb58582; no voters:
I20260619 20:49:37.344385 2719 raft_consensus.cc:2781] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:37.344527 2719 raft_consensus.cc:686] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff2e74f36294ac895490dd33eb58582, State: Running, Role: LEADER
I20260619 20:49:37.344630 2719 consensus_queue.cc:237] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.345400 2123 catalog_manager.cc:5495] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5ff2e74f36294ac895490dd33eb58582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } health_report { overall_health: UNKNOWN } } }
I20260619 20:49:37.390856 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.388214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.98.1:35295)
0619 20:49:37.388373 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.388375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.388382 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:37.388416 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.388417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.388436 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.388508 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:37.388633 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.389136 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.390002 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.390198 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.390249 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.390490 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:37.390596 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:37.390653 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.390768 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.390783 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260619 20:49:37.395916 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.393407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42387 (local address 127.0.98.2:36931)
0619 20:49:37.393448 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.393450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.393458 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:37.393500 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.393501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.393512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.393565 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:37.393714 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.394340 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.395122 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.395382 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.395472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.395650 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:37.395683 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:37.395769 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.395826 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.395846 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:49:37.395936 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.393194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.98.1:42387)
0619 20:49:37.393380 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:37.393391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:37.393443 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:37.393573 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:37.393574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:37.393576 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:37.393632 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.393634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.394420 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.394421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.395037 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.395039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.395471 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.395472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.395538 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.395558 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:37.395770 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:37.395772 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:37.395822 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:49:37.395861 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
I20260619 20:49:37.398097 2560 raft_consensus.cc:1261] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Refusing update from remote peer 5ff2e74f36294ac895490dd33eb58582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:37.398087 2417 raft_consensus.cc:1261] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Refusing update from remote peer 5ff2e74f36294ac895490dd33eb58582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:37.398372 2719 consensus_queue.cc:1035] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:37.398526 2725 consensus_queue.cc:1035] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:37.405475 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.403070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60263 (local address 127.0.98.1:35295)
0619 20:49:37.403109 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.403111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.403213 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:49:37.403277 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.403277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.403294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.403331 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:37.403420 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.403844 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.404514 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.404747 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.404806 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.405193 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:37.405223 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:37.405250 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.405395 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.405414 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:37.405509 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.403017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35295 (local address 127.0.98.2:60263)
0619 20:49:37.403133 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:37.403143 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:37.403215 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:37.403339 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:37.403340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:37.403341 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:49:37.403381 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.403382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.403900 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.403901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.404445 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:37.404446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.405038 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.405039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:37.405100 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.405105 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:49:37.405332 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:49:37.405334 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:37.405380 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:49:37.405432 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260619 20:49:37.415036 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:37.415162 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:37.415856 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Sending DeleteTablet for 3 replicas of tablet d2102c46ce294915b3e2aa62fce083af
I20260619 20:49:37.416174 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet d2102c46ce294915b3e2aa62fce083af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39592
I20260619 20:49:37.416188 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet d2102c46ce294915b3e2aa62fce083af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47454
I20260619 20:49:37.416298 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet d2102c46ce294915b3e2aa62fce083af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56796
I20260619 20:49:37.416320 2704 tablet_replica.cc:330] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: stopping tablet replica
I20260619 20:49:37.416385 2705 tablet_replica.cc:330] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: stopping tablet replica
I20260619 20:49:37.416406 2704 raft_consensus.cc:2229] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:37.416427 2705 raft_consensus.cc:2229] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:37.416450 2704 raft_consensus.cc:2258] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:37.416502 2705 raft_consensus.cc:2258] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:37.416543 2703 tablet_replica.cc:330] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: stopping tablet replica
I20260619 20:49:37.416607 2703 raft_consensus.cc:2229] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:37.416651 2703 raft_consensus.cc:2258] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:37.416822 2704 ts_tablet_manager.cc:1905] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:37.416986 2703 ts_tablet_manager.cc:1905] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:37.416991 2705 ts_tablet_manager.cc:1905] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:37.418743 2704 ts_tablet_manager.cc:1918] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:37.418787 2705 ts_tablet_manager.cc:1918] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:37.418807 2704 log.cc:1196] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/d2102c46ce294915b3e2aa62fce083af
I20260619 20:49:37.418838 2705 log.cc:1196] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/d2102c46ce294915b3e2aa62fce083af
I20260619 20:49:37.418826 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:37.418982 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:37.419054 2705 ts_tablet_manager.cc:1939] T d2102c46ce294915b3e2aa62fce083af P 5ff2e74f36294ac895490dd33eb58582: Deleting consensus metadata
I20260619 20:49:37.419054 2704 ts_tablet_manager.cc:1939] T d2102c46ce294915b3e2aa62fce083af P 2584614f312f44e68913f7f5e2714b3d: Deleting consensus metadata
I20260619 20:49:37.419401 2703 ts_tablet_manager.cc:1918] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:49:37.419445 2703 log.cc:1196] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/d2102c46ce294915b3e2aa62fce083af
I20260619 20:49:37.419732 2703 ts_tablet_manager.cc:1939] T d2102c46ce294915b3e2aa62fce083af P 4acf21755b2f42908e8b769e14a6f143: Deleting consensus metadata
I20260619 20:49:37.419839 2109 catalog_manager.cc:4841] TS 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341): tablet d2102c46ce294915b3e2aa62fce083af (table test-table [id=bc6a4a5e5f7b41c5855f59d44c9957a2]) successfully deleted
I20260619 20:49:37.419904 2109 catalog_manager.cc:4841] TS 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295): tablet d2102c46ce294915b3e2aa62fce083af (table test-table [id=bc6a4a5e5f7b41c5855f59d44c9957a2]) successfully deleted
I20260619 20:49:37.420037 2107 catalog_manager.cc:4841] TS 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931): tablet d2102c46ce294915b3e2aa62fce083af (table test-table [id=bc6a4a5e5f7b41c5855f59d44c9957a2]) successfully deleted
I20260619 20:49:37.420773 2260 tablet_service.cc:1467] Processing CreateTablet for tablet b5e946f0a62b4388b518c2ab6d578167 (DEFAULT_TABLE table=test-table [id=b2ef8c5c32d040e3a94551a4a6f3cbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:37.420775 2540 tablet_service.cc:1467] Processing CreateTablet for tablet b5e946f0a62b4388b518c2ab6d578167 (DEFAULT_TABLE table=test-table [id=b2ef8c5c32d040e3a94551a4a6f3cbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:37.420876 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e946f0a62b4388b518c2ab6d578167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:37.420878 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e946f0a62b4388b518c2ab6d578167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:37.421793 2718 tablet_bootstrap.cc:492] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Bootstrap starting.
I20260619 20:49:37.421888 2400 tablet_service.cc:1467] Processing CreateTablet for tablet b5e946f0a62b4388b518c2ab6d578167 (DEFAULT_TABLE table=test-table [id=b2ef8c5c32d040e3a94551a4a6f3cbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:37.421986 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e946f0a62b4388b518c2ab6d578167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:37.422111 2718 tablet_bootstrap.cc:654] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:37.422761 2718 tablet_bootstrap.cc:492] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: No bootstrap required, opened a new log
I20260619 20:49:37.422820 2718 ts_tablet_manager.cc:1397] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:37.422924 2718 raft_consensus.cc:348] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.422999 2718 raft_consensus.cc:374] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:37.423029 2718 raft_consensus.cc:729] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Initialized, Role: FOLLOWER
I20260619 20:49:37.423084 2718 consensus_queue.cc:260] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.423206 2718 ts_tablet_manager.cc:1428] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:37.423264 2715 tablet_bootstrap.cc:492] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582: Bootstrap starting.
I20260619 20:49:37.423614 2715 tablet_bootstrap.cc:654] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:37.423902 2714 tablet_bootstrap.cc:492] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Bootstrap starting.
I20260619 20:49:37.424192 2714 tablet_bootstrap.cc:654] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:37.424764 2714 tablet_bootstrap.cc:492] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: No bootstrap required, opened a new log
I20260619 20:49:37.424811 2714 ts_tablet_manager.cc:1397] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:37.424890 2714 raft_consensus.cc:348] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.424943 2714 raft_consensus.cc:374] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:37.424957 2714 raft_consensus.cc:729] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf21755b2f42908e8b769e14a6f143, State: Initialized, Role: FOLLOWER
I20260619 20:49:37.425019 2714 consensus_queue.cc:260] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.425153 2714 ts_tablet_manager.cc:1428] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:37.425467 2715 tablet_bootstrap.cc:492] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582: No bootstrap required, opened a new log
I20260619 20:49:37.425523 2715 ts_tablet_manager.cc:1397] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:37.425730 2715 raft_consensus.cc:348] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.425812 2715 raft_consensus.cc:374] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:37.425835 2715 raft_consensus.cc:729] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff2e74f36294ac895490dd33eb58582, State: Initialized, Role: FOLLOWER
I20260619 20:49:37.425875 2715 consensus_queue.cc:260] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.426016 2715 ts_tablet_manager.cc:1428] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:37.583984 2632 raft_consensus.cc:482] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:37.584086 2632 raft_consensus.cc:504] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.584273 2632 leader_election.cc:288] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295), 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931)
I20260619 20:49:37.584512 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e946f0a62b4388b518c2ab6d578167" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582" is_pre_election: true
I20260619 20:49:37.584512 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e946f0a62b4388b518c2ab6d578167" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf21755b2f42908e8b769e14a6f143" is_pre_election: true
I20260619 20:49:37.584605 2280 raft_consensus.cc:2454] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 0.
I20260619 20:49:37.584605 2417 raft_consensus.cc:2454] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 0.
I20260619 20:49:37.584808 2493 leader_election.cc:302] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2584614f312f44e68913f7f5e2714b3d, 5ff2e74f36294ac895490dd33eb58582; no voters:
I20260619 20:49:37.584890 2632 raft_consensus.cc:2781] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:37.584918 2632 raft_consensus.cc:482] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:37.584936 2632 raft_consensus.cc:3037] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:37.585625 2632 raft_consensus.cc:504] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.585752 2632 leader_election.cc:288] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 election: Requested vote from peers 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295), 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931)
I20260619 20:49:37.585901 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e946f0a62b4388b518c2ab6d578167" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf21755b2f42908e8b769e14a6f143"
I20260619 20:49:37.585971 2417 raft_consensus.cc:3037] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:37.585922 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e946f0a62b4388b518c2ab6d578167" candidate_uuid: "2584614f312f44e68913f7f5e2714b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff2e74f36294ac895490dd33eb58582"
I20260619 20:49:37.586042 2280 raft_consensus.cc:3037] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:37.586582 2417 raft_consensus.cc:2454] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 1.
I20260619 20:49:37.586737 2280 raft_consensus.cc:2454] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2584614f312f44e68913f7f5e2714b3d in term 1.
I20260619 20:49:37.586768 2491 leader_election.cc:302] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2584614f312f44e68913f7f5e2714b3d, 4acf21755b2f42908e8b769e14a6f143; no voters:
I20260619 20:49:37.586843 2632 raft_consensus.cc:2781] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:37.586889 2632 raft_consensus.cc:686] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 1 LEADER]: Becoming Leader. State: Replica: 2584614f312f44e68913f7f5e2714b3d, State: Running, Role: LEADER
I20260619 20:49:37.586937 2632 consensus_queue.cc:237] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } }
I20260619 20:49:37.587577 2122 catalog_manager.cc:5495] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2584614f312f44e68913f7f5e2714b3d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2584614f312f44e68913f7f5e2714b3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2584614f312f44e68913f7f5e2714b3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39341 } health_report { overall_health: HEALTHY } } }
I20260619 20:49:37.610924 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:37.609018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.98.3:39341)
0619 20:49:37.609081 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:37.609083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:37.609089 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:49:37.609107 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:37.609107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:37.609117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:37.609160 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:37.609250 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.609756 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.610430 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:37.610604 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:37.610652 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:37.610756 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:37.610777 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:37.610810 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:37.610861 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:49:37.610876 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:49:37.612869 2417 raft_consensus.cc:1261] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Refusing update from remote peer 2584614f312f44e68913f7f5e2714b3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:37.612946 2280 raft_consensus.cc:1261] T b5e946f0a62b4388b518c2ab6d578167 P 5ff2e74f36294ac895490dd33eb58582 [term 1 FOLLOWER]: Refusing update from remote peer 2584614f312f44e68913f7f5e2714b3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:37.613087 2632 consensus_queue.cc:1035] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acf21755b2f42908e8b769e14a6f143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:37.613158 2695 consensus_queue.cc:1035] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff2e74f36294ac895490dd33eb58582" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:49:37.617194 2686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 20:49:39.133141 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:39.133237 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:39.134012 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44adc2809d744d90a345a6bdf3f52f28: Sending DeleteTablet for 3 replicas of tablet b5e946f0a62b4388b518c2ab6d578167
I20260619 20:49:39.134241 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e946f0a62b4388b518c2ab6d578167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56796
I20260619 20:49:39.134268 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e946f0a62b4388b518c2ab6d578167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47454
I20260619 20:49:39.134416 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e946f0a62b4388b518c2ab6d578167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39592
I20260619 20:49:39.134418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 2195
I20260619 20:49:39.135105 2769 tablet_replica.cc:330] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: stopping tablet replica
I20260619 20:49:39.135176 2769 raft_consensus.cc:2229] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:39.135221 2769 raft_consensus.cc:2258] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:39.135439 2767 tablet_replica.cc:330] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: stopping tablet replica
I20260619 20:49:39.135466 2769 ts_tablet_manager.cc:1905] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:39.135502 2767 raft_consensus.cc:2229] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:39.135561 2767 raft_consensus.cc:2258] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:39.135794 2767 ts_tablet_manager.cc:1905] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:39.137130 2769 ts_tablet_manager.cc:1918] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:49:39.137199 2769 log.cc:1196] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/b5e946f0a62b4388b518c2ab6d578167
I20260619 20:49:39.137411 2767 ts_tablet_manager.cc:1918] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:49:39.137447 2769 ts_tablet_manager.cc:1939] T b5e946f0a62b4388b518c2ab6d578167 P 4acf21755b2f42908e8b769e14a6f143: Deleting consensus metadata
I20260619 20:49:39.137475 2767 log.cc:1196] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/b5e946f0a62b4388b518c2ab6d578167
I20260619 20:49:39.137709 2767 ts_tablet_manager.cc:1939] T b5e946f0a62b4388b518c2ab6d578167 P 2584614f312f44e68913f7f5e2714b3d: Deleting consensus metadata
I20260619 20:49:39.137831 2107 catalog_manager.cc:4841] TS 4acf21755b2f42908e8b769e14a6f143 (127.0.98.2:36931): tablet b5e946f0a62b4388b518c2ab6d578167 (table test-table [id=b2ef8c5c32d040e3a94551a4a6f3cbc1]) successfully deleted
I20260619 20:49:39.137985 2109 catalog_manager.cc:4841] TS 2584614f312f44e68913f7f5e2714b3d (127.0.98.3:39341): tablet b5e946f0a62b4388b518c2ab6d578167 (table test-table [id=b2ef8c5c32d040e3a94551a4a6f3cbc1]) successfully deleted
W20260619 20:49:39.141606 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35295 (error 108)
I20260619 20:49:39.141844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 2335
I20260619 20:49:39.145479 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:39.141941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35295 (local address 127.0.0.1:56812)
0619 20:49:39.145400 (+ 3459us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:39.145447 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3433,"thread_start_us":65,"threads_started":1}
W20260619 20:49:39.145617 2109 catalog_manager.cc:4584] TS 5ff2e74f36294ac895490dd33eb58582 (127.0.98.1:35295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5e946f0a62b4388b518c2ab6d578167: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35295: connect: Connection refused (error 111)
I20260619 20:49:39.148363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 2475
I20260619 20:49:39.153669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 2091
2026-06-19T20:49:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5777 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:39 dist-test-slave-0smx krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:39 dist-test-slave-0smx krb5kdc[2774](info): set up 2 sockets
Jun 19 20:49:39 dist-test-slave-0smx krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260619 20:49:40.189064 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:40.300055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35757
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:40.378739 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:40.378976 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:40.379015 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:40.380164 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:40.380234 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:40.380257 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:40.380285 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:40.380316 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:40.381824 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:40.382122 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:40.382354 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:40.387056 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:40.387122 2787 server_base.cc:1034] running on GCE node
W20260619 20:49:40.387207 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.387072 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:40.387470 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:40.387749 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:40.388911 2787 hybrid_clock.cc:648] HybridClock initialized: now 1781902180388899 us; error 42 us; skew 500 ppm
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:40.391239 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:40.391637 2787 webserver.cc:460] Webserver started at http://127.0.98.62:46519/ using document root <none> and password file <none>
I20260619 20:49:40.391863 2787 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:40.391928 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:40.392087 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:40.393088 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "f0e183bb384b4e05a61d4066d380c88d"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "ee7c607f14393b9f9a7aa6581eac1c21"
server_key_iv: "07b4fe527a03c4ee522effef82a349d5"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.393479 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0e183bb384b4e05a61d4066d380c88d"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "ee7c607f14393b9f9a7aa6581eac1c21"
server_key_iv: "07b4fe527a03c4ee522effef82a349d5"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.394735 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:40.395447 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.395615 2787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:40.395709 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "f0e183bb384b4e05a61d4066d380c88d"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "ee7c607f14393b9f9a7aa6581eac1c21"
server_key_iv: "07b4fe527a03c4ee522effef82a349d5"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.395788 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:40.409603 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:40.411267 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:40.411420 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:40.415329 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35757
I20260619 20:49:40.415375 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35757 every 8 connection(s)
I20260619 20:49:40.415876 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:40.416692 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:40.419416 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d: Bootstrap starting.
I20260619 20:49:40.419991 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:40.420277 2855 log.cc:824] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:40.420923 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d: No bootstrap required, opened a new log
I20260619 20:49:40.422484 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e183bb384b4e05a61d4066d380c88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35757 } }
I20260619 20:49:40.422647 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:40.422694 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e183bb384b4e05a61d4066d380c88d, State: Initialized, Role: FOLLOWER
I20260619 20:49:40.422830 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e183bb384b4e05a61d4066d380c88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35757 } }
I20260619 20:49:40.422906 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:40.422950 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:40.422999 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:40.423825 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e183bb384b4e05a61d4066d380c88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35757 } }
I20260619 20:49:40.423938 2855 leader_election.cc:302] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e183bb384b4e05a61d4066d380c88d; no voters:
I20260619 20:49:40.424132 2855 leader_election.cc:288] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:40.424175 2858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:40.424319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 2787
I20260619 20:49:40.424345 2858 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [term 1 LEADER]: Becoming Leader. State: Replica: f0e183bb384b4e05a61d4066d380c88d, State: Running, Role: LEADER
I20260619 20:49:40.424408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:40.424415 2858 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e183bb384b4e05a61d4066d380c88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35757 } }
I20260619 20:49:40.424583 392 external_mini_cluster.cc:1428] Setting key c4564a553e1311b5b0508c723486360b
I20260619 20:49:40.424434 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:40.424710 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e183bb384b4e05a61d4066d380c88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e183bb384b4e05a61d4066d380c88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35757 } } }
I20260619 20:49:40.424773 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:40.424911 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e183bb384b4e05a61d4066d380c88d. Latest consensus state: current_term: 1 leader_uuid: "f0e183bb384b4e05a61d4066d380c88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e183bb384b4e05a61d4066d380c88d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35757 } } }
I20260619 20:49:40.424962 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d [sys.catalog]: This master's current role is: LEADER
W20260619 20:49:40.426321 2876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:49:40.426376 2876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:49:40.426433 2862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:40.426656 2862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:40.428252 2862 catalog_manager.cc:1347] Generated new cluster ID: 967ca6e9b4244fc2aea2816829a13582
I20260619 20:49:40.428305 2862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:40.436699 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.425151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.0.98.62:35757)
0619 20:49:40.425376 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:40.425385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:40.426153 (+ 768us) server_negotiation.cc:407] Connection header received
0619 20:49:40.426273 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:40.426281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:40.426337 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:40.426543 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:40.427317 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.428558 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.429636 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.430247 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.430347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.433032 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:40.433036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:40.433036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:40.433046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:40.434960 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.435316 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.435319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.435319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.435408 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.435572 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.435572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.435573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.435855 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:40.436311 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:40.436373 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:49:40.436492 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260619 20:49:40.440862 2862 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:40.441427 2862 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:40.446116 2862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e183bb384b4e05a61d4066d380c88d: Generated new TSK 0
I20260619 20:49:40.446272 2862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:40.472358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35757
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:40.550782 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:40.550977 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:40.551000 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:40.552172 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:40.552211 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:40.552237 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:40.553745 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:40.553959 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:40.554201 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:40.559243 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.559268 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.559243 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:40.559298 2881 server_base.cc:1034] running on GCE node
I20260619 20:49:40.559684 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:40.559942 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:40.561082 2881 hybrid_clock.cc:648] HybridClock initialized: now 1781902180561073 us; error 31 us; skew 500 ppm
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:40.563513 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:40.563936 2881 webserver.cc:460] Webserver started at http://127.0.98.1:36613/ using document root <none> and password file <none>
I20260619 20:49:40.564152 2881 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:40.564217 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:40.564361 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:40.565426 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5aa51f6de5340309be6428338b0aea1"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "3137637341a9db60165359bc68b0f8e1"
server_key_iv: "4b15391f55f736aed4c63e3651b56b5f"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.565773 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5aa51f6de5340309be6428338b0aea1"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "3137637341a9db60165359bc68b0f8e1"
server_key_iv: "4b15391f55f736aed4c63e3651b56b5f"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.567094 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:40.567829 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.568125 2881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:40.568229 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "f5aa51f6de5340309be6428338b0aea1"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "3137637341a9db60165359bc68b0f8e1"
server_key_iv: "4b15391f55f736aed4c63e3651b56b5f"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.568300 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:40.578491 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:40.580125 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:40.580277 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:40.580479 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:40.580796 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:40.580852 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.580890 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:40.580917 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.586985 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40819
I20260619 20:49:40.587038 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40819 every 8 connection(s)
I20260619 20:49:40.587364 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:40.596161 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.588056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39699 (local address 127.0.98.62:35757)
0619 20:49:40.588119 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:40.588121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:40.588305 (+ 184us) server_negotiation.cc:407] Connection header received
0619 20:49:40.589264 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:40.589265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:40.589277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:40.589373 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:40.589975 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.590480 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.591459 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.591676 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.591762 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.593673 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:40.593675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:40.593676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:40.593685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:40.594866 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.595257 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.595258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.595259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.595330 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.595558 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.595559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.595559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.595667 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:40.595742 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:40.595985 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 20:49:40.596061 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:49:40.596318 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.588043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35757 (local address 127.0.98.1:39699)
0619 20:49:40.588197 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:40.588229 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:40.589149 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:40.589446 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:40.589449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:40.589684 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:40.589870 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:40.589873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.590568 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.590569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:40.591354 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:40.591356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.591770 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.591771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:40.591886 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.592283 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:40.592290 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:40.593565 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:40.594959 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:40.594965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:40.594966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:40.595169 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:40.595393 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:40.595394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:40.595395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:40.595492 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:40.595732 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:40.595737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:40.595863 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 20:49:40.595965 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 20:49:40.596106 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260619 20:49:40.596555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 2881
I20260619 20:49:40.596638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:40.596854 392 external_mini_cluster.cc:1428] Setting key 1b1d49596b83f14a3c797396429ad2cb
I20260619 20:49:40.596869 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35757
I20260619 20:49:40.596949 3010 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:40.597110 3010 heartbeater.cc:511] Master 127.0.98.62:35757 requested a full tablet report, sending...
I20260619 20:49:40.597690 2819 ts_manager.cc:194] Registered new tserver with Master: f5aa51f6de5340309be6428338b0aea1 (127.0.98.1:40819)
I20260619 20:49:40.598356 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39699
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:40.620551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35757
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:40.700129 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:40.700313 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:40.700336 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:40.701505 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:40.701545 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:40.701594 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:40.703042 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:40.703271 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:40.703492 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:40.708384 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.708367 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.708367 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:40.708398 3015 server_base.cc:1034] running on GCE node
I20260619 20:49:40.708772 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:40.708982 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:40.710124 3015 hybrid_clock.cc:648] HybridClock initialized: now 1781902180710121 us; error 34 us; skew 500 ppm
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:40.712428 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:40.712858 3015 webserver.cc:460] Webserver started at http://127.0.98.2:46405/ using document root <none> and password file <none>
I20260619 20:49:40.713032 3015 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:40.713070 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:40.713161 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:40.714128 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "b509d574621845478bc8225f60a01497"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "dfb7cc2f2086b0e86fbcd069aed942a8"
server_key_iv: "eb3a3fb93abd0b75e8453b43948ad192"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.714480 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b509d574621845478bc8225f60a01497"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "dfb7cc2f2086b0e86fbcd069aed942a8"
server_key_iv: "eb3a3fb93abd0b75e8453b43948ad192"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.715718 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:40.716353 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.716557 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:40.716646 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "b509d574621845478bc8225f60a01497"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "dfb7cc2f2086b0e86fbcd069aed942a8"
server_key_iv: "eb3a3fb93abd0b75e8453b43948ad192"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.716723 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:40.730064 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:40.731699 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:40.731848 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:40.732082 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:40.732360 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:40.732417 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.732465 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:40.732492 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.738209 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38701
I20260619 20:49:40.738260 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38701 every 8 connection(s)
I20260619 20:49:40.738579 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:40.744747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3015
I20260619 20:49:40.744840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:40.745059 392 external_mini_cluster.cc:1428] Setting key f59de6050aac9ac24596fa4384f36882
I20260619 20:49:40.747900 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.739291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52341 (local address 127.0.98.62:35757)
0619 20:49:40.739355 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:40.739357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:40.739537 (+ 180us) server_negotiation.cc:407] Connection header received
0619 20:49:40.740457 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:40.740458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:40.740470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:40.740524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:40.741116 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.741691 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.742655 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.742870 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.742955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.744993 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:40.744994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:40.744995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:40.745001 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:40.746535 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.746948 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.746950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.746951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.747012 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.747276 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.747277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.747277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.747389 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:40.747465 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:40.747733 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 20:49:40.747812 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:49:40.748080 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.739274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35757 (local address 127.0.98.2:52341)
0619 20:49:40.739424 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:40.739456 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:40.740348 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:40.740588 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:40.740592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:40.740840 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:40.741012 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:40.741016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.741779 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.741780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:40.742564 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:40.742566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.742962 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.742963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:40.743082 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.743503 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:40.743511 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:40.744906 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:40.746635 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:40.746641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:40.746643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:40.746858 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:40.747080 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:40.747081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:40.747081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:40.747201 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:40.747457 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:40.747462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:40.747598 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 20:49:40.747700 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 20:49:40.747847 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260619 20:49:40.748699 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35757
I20260619 20:49:40.748798 3144 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:40.748979 3144 heartbeater.cc:511] Master 127.0.98.62:35757 requested a full tablet report, sending...
I20260619 20:49:40.749356 2819 ts_manager.cc:194] Registered new tserver with Master: b509d574621845478bc8225f60a01497 (127.0.98.2:38701)
I20260619 20:49:40.749686 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52341
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:40.769348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35757
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:40.848400 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:40.848594 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:40.848616 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:40.849694 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:40.849727 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:40.849751 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:40.851257 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:40.851423 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:40.851599 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:40.856491 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.856491 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:40.856561 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:40.856602 3149 server_base.cc:1034] running on GCE node
I20260619 20:49:40.857002 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:40.857262 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:40.858469 3149 hybrid_clock.cc:648] HybridClock initialized: now 1781902180858448 us; error 35 us; skew 500 ppm
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:40.860888 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:40.861253 3149 webserver.cc:460] Webserver started at http://127.0.98.3:34785/ using document root <none> and password file <none>
I20260619 20:49:40.861512 3149 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:40.861558 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:40.861650 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:40.862591 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0cdcc2aab014bf7b89d86dabdcda7a8"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "25b0239d1b6170be399b52e46cc24eeb"
server_key_iv: "0beb82266c3d110b1470af2556fdd576"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.862867 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0cdcc2aab014bf7b89d86dabdcda7a8"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "25b0239d1b6170be399b52e46cc24eeb"
server_key_iv: "0beb82266c3d110b1470af2556fdd576"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.864095 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:49:40.864713 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.864866 3149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:40.864961 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "e0cdcc2aab014bf7b89d86dabdcda7a8"
format_stamp: "Formatted at 2026-06-19 20:49:40 on dist-test-slave-0smx"
server_key: "25b0239d1b6170be399b52e46cc24eeb"
server_key_iv: "0beb82266c3d110b1470af2556fdd576"
server_key_version: "encryptionkey@0"
I20260619 20:49:40.865041 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:40.882633 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:40.884428 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:40.884578 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:40.884790 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:40.885095 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:40.885149 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.885188 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:40.885218 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:40.890939 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41435
I20260619 20:49:40.890988 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41435 every 8 connection(s)
I20260619 20:49:40.891287 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:40.893641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3149
I20260619 20:49:40.893743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:40.893934 392 external_mini_cluster.cc:1428] Setting key 0f9a09b7314b5a9413b178ce46e864c1
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:40.900496 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.891971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35253 (local address 127.0.98.62:35757)
0619 20:49:40.892063 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:40.892066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:40.892254 (+ 188us) server_negotiation.cc:407] Connection header received
0619 20:49:40.893132 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:40.893133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:40.893142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:40.893193 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:40.894036 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.894737 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.895817 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.896027 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.896110 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.898137 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:40.898138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:40.898138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:40.898144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:40.899423 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.899771 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.899772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.899772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.899819 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:40.900019 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:40.900020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:40.900020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:40.900118 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:40.900165 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:40.900380 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 20:49:40.900436 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 20:49:40.900734 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:40.891941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35757 (local address 127.0.98.3:35253)
0619 20:49:40.892135 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:40.892170 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:40.893045 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:40.893260 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:40.893263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:40.893684 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:40.893935 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:40.893940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.894806 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.894808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:40.895717 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:40.895719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:40.896117 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:40.896118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:40.896246 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:40.896657 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:40.896665 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:40.898058 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:40.899491 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:40.899496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:40.899498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:40.899704 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:40.899875 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:40.899876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:40.899876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:40.899968 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:40.900171 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:40.900175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:40.900276 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:49:40.900353 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:49:40.900481 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":123,"threads_started":1}
I20260619 20:49:40.901173 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35757
I20260619 20:49:40.901279 3278 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:40.901473 3278 heartbeater.cc:511] Master 127.0.98.62:35757 requested a full tablet report, sending...
I20260619 20:49:40.901845 2819 ts_manager.cc:194] Registered new tserver with Master: e0cdcc2aab014bf7b89d86dabdcda7a8 (127.0.98.3:41435)
I20260619 20:49:40.902207 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35253
I20260619 20:49:40.905671 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:49:40.915916 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:40.915121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.0.98.62:35757)
0619 20:49:40.915180 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:40.915186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:40.915299 (+ 113us) server_negotiation.cc:407] Connection header received
0619 20:49:40.915354 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:40.915354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:40.915362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:40.915412 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:40.915869 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39336: BlockingRecv error: recv got EOF from 127.0.0.1:39336 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:49:40.916834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 2881
I20260619 20:49:40.921068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3015
I20260619 20:49:40.925421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3149
I20260619 20:49:40.929790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 2787
2026-06-19T20:49:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1775 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[3290](info): set up 2 sockets
Jun 19 20:49:40 dist-test-slave-0smx krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260619 20:49:41.957015 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:42 dist-test-slave-0smx krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:42.069084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:42.146126 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:42.146310 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:42.146332 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:42.147459 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:42.147498 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:42.147513 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:42.147526 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:42.147576 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:42.148993 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:42.149219 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:42.149437 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:42.153998 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:42.154018 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:42.154033 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:42.154241 3303 server_base.cc:1034] running on GCE node
I20260619 20:49:42.154474 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:42.154753 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:42.155915 3303 hybrid_clock.cc:648] HybridClock initialized: now 1781902182155901 us; error 45 us; skew 500 ppm
Jun 19 20:49:42 dist-test-slave-0smx krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:42.160823 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:42.161276 3303 webserver.cc:460] Webserver started at http://127.0.98.62:44351/ using document root <none> and password file <none>
I20260619 20:49:42.161562 3303 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:42.161630 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:42.161783 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:42.162779 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "83891b5a11bc4530b2bbbc85ebf095a6"
format_stamp: "Formatted at 2026-06-19 20:49:42 on dist-test-slave-0smx"
server_key: "d2477054cd8f9dd491c14ab1b268be8a"
server_key_iv: "662c5bb8cd45019708b7949406305153"
server_key_version: "encryptionkey@0"
I20260619 20:49:42.163131 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "83891b5a11bc4530b2bbbc85ebf095a6"
format_stamp: "Formatted at 2026-06-19 20:49:42 on dist-test-slave-0smx"
server_key: "d2477054cd8f9dd491c14ab1b268be8a"
server_key_iv: "662c5bb8cd45019708b7949406305153"
server_key_version: "encryptionkey@0"
I20260619 20:49:42.164379 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:42.165058 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:42.165230 3303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:42.165345 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "83891b5a11bc4530b2bbbc85ebf095a6"
format_stamp: "Formatted at 2026-06-19 20:49:42 on dist-test-slave-0smx"
server_key: "d2477054cd8f9dd491c14ab1b268be8a"
server_key_iv: "662c5bb8cd45019708b7949406305153"
server_key_version: "encryptionkey@0"
I20260619 20:49:42.165428 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:42.176376 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:42.178186 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:42.178339 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:42.182372 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34601
I20260619 20:49:42.182422 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34601 every 8 connection(s)
I20260619 20:49:42.182765 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:42.182948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3303
I20260619 20:49:42.183053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:42.183249 392 external_mini_cluster.cc:1428] Setting key f86d5a7ee7a5b7febbeb609b984294a0
I20260619 20:49:42.183785 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:42.186717 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6: Bootstrap starting.
I20260619 20:49:42.187407 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:42.187685 3371 log.cc:824] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:42.188446 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6: No bootstrap required, opened a new log
Jun 19 20:49:42 dist-test-slave-0smx krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:42.190742 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34601 } }
I20260619 20:49:42.190915 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:42.190951 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83891b5a11bc4530b2bbbc85ebf095a6, State: Initialized, Role: FOLLOWER
I20260619 20:49:42.191087 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34601 } }
I20260619 20:49:42.191164 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:42.191186 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:42.191248 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:42.191924 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34601 } }
I20260619 20:49:42.192034 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83891b5a11bc4530b2bbbc85ebf095a6; no voters:
I20260619 20:49:42.192186 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:42.192253 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:42.192431 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:42.192459 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [term 1 LEADER]: Becoming Leader. State: Replica: 83891b5a11bc4530b2bbbc85ebf095a6, State: Running, Role: LEADER
I20260619 20:49:42.192545 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34601 } }
I20260619 20:49:42.192948 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83891b5a11bc4530b2bbbc85ebf095a6. Latest consensus state: current_term: 1 leader_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34601 } } }
I20260619 20:49:42.193086 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:42.193058 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83891b5a11bc4530b2bbbc85ebf095a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34601 } } }
I20260619 20:49:42.193230 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:42.193778 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:42.194058 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:42.183724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.0.98.62:34601)
0619 20:49:42.183921 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:42.183929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:42.183952 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:49:42.184379 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:42.184385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:42.184429 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:42.184537 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:42.185375 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:42.186249 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:42.187286 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:42.187729 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:42.187854 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:42.190252 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:42.190257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:42.190257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:42.190270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:42.192258 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:42.192658 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:42.192661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:42.192662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:42.192758 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:42.192994 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:42.192995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:42.192995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:42.193345 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:42.193470 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:42.193735 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 20:49:42.193898 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260619 20:49:42.194414 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:42.195211 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:49:42.195868 3385 catalog_manager.cc:1347] Generated new cluster ID: a52058bedba74358b7aa305fe12d24dd
I20260619 20:49:42.195919 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:42.208426 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:42.205133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.0.98.62:34601)
0619 20:49:42.205186 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:42.205188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:42.205329 (+ 141us) server_negotiation.cc:407] Connection header received
0619 20:49:42.205411 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:42.205412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:42.205422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:42.205473 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:42.205832 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:42.206288 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:42.207037 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:42.207242 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:42.207326 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:42.207987 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:42.207988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:42.207988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:42.208163 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:42.208177 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:42.208182 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:42.208249 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:42.208318 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 20:49:42.208338 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:49:42.214488 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:42.215103 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:42.219341 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83891b5a11bc4530b2bbbc85ebf095a6: Generated new TSK 0
I20260619 20:49:42.219518 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:49:42.224421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3303
2026-06-19T20:49:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1291 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:42 dist-test-slave-0smx krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:42 dist-test-slave-0smx krb5kdc[3404](info): set up 2 sockets
Jun 19 20:49:42 dist-test-slave-0smx krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260619 20:49:43.249261 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:43.363725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:43.444059 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:43.444267 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:43.444290 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:43.445484 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:43.445525 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:43.445539 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:43.445552 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:43.445564 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:43.447039 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:43.447280 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:43.447510 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:43.452108 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:43.452080 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:43.452216 3417 server_base.cc:1034] running on GCE node
W20260619 20:49:43.452075 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:43.452441 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:43.452759 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:43.453936 3417 hybrid_clock.cc:648] HybridClock initialized: now 1781902183453912 us; error 35 us; skew 500 ppm
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:43.456385 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:43.456864 3417 webserver.cc:460] Webserver started at http://127.0.98.62:36337/ using document root <none> and password file <none>
I20260619 20:49:43.457080 3417 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:43.457147 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:43.457346 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:43.458390 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "a75d5a0eca2a4a2d91183f20400ec14d"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "03fc5755fbfa25a5c118233cb1fd6d90"
server_key_iv: "1c2b8af56af2f7b694fae42e8b81584c"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.458746 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "a75d5a0eca2a4a2d91183f20400ec14d"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "03fc5755fbfa25a5c118233cb1fd6d90"
server_key_iv: "1c2b8af56af2f7b694fae42e8b81584c"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.460029 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:49:43.460788 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.460968 3417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:43.461063 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "a75d5a0eca2a4a2d91183f20400ec14d"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "03fc5755fbfa25a5c118233cb1fd6d90"
server_key_iv: "1c2b8af56af2f7b694fae42e8b81584c"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.461148 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:43.479806 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:43.481945 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:43.482105 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:43.486075 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35249
I20260619 20:49:43.486110 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35249 every 8 connection(s)
I20260619 20:49:43.486487 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:43.487473 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:43.487905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3417
I20260619 20:49:43.488015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:43.488184 392 external_mini_cluster.cc:1428] Setting key 29d67d7fd1d00f8feb3209169bd747ba
I20260619 20:49:43.490567 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d: Bootstrap starting.
I20260619 20:49:43.491233 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:43.491523 3485 log.cc:824] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:43.492158 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d: No bootstrap required, opened a new log
I20260619 20:49:43.493918 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260619 20:49:43.494043 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:43.494065 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75d5a0eca2a4a2d91183f20400ec14d, State: Initialized, Role: FOLLOWER
I20260619 20:49:43.494197 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260619 20:49:43.494269 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:43.494290 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:43.494362 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:43.494856 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260619 20:49:43.494972 3485 leader_election.cc:302] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a75d5a0eca2a4a2d91183f20400ec14d; no voters:
I20260619 20:49:43.495151 3485 leader_election.cc:288] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:43.495500 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:43.495182 3490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:43.496112 3490 raft_consensus.cc:686] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [term 1 LEADER]: Becoming Leader. State: Replica: a75d5a0eca2a4a2d91183f20400ec14d, State: Running, Role: LEADER
I20260619 20:49:43.496232 3490 consensus_queue.cc:237] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260619 20:49:43.496739 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a75d5a0eca2a4a2d91183f20400ec14d. Latest consensus state: current_term: 1 leader_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } } }
I20260619 20:49:43.496835 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:43.496955 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75d5a0eca2a4a2d91183f20400ec14d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } } }
I20260619 20:49:43.497011 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:43.497131 3500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:43.497606 3500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:43.499127 3500 catalog_manager.cc:1347] Generated new cluster ID: def0c50766cc4dd48c2f4c252688827e
I20260619 20:49:43.499178 3500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:43.500887 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.488709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.98.62:35249)
0619 20:49:43.488908 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:43.488918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:43.488952 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:49:43.489518 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:43.489526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:43.489588 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:43.489755 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:43.490340 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.491710 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.492547 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.492979 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.493099 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.495753 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:43.495758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:43.495758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:43.495772 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:43.499343 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.499714 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.499717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.499718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.499819 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.500012 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.500013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.500014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.500303 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:43.500450 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:43.500544 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 20:49:43.500677 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260619 20:49:43.511209 3500 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:43.511897 3500 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:43.523473 3500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a75d5a0eca2a4a2d91183f20400ec14d: Generated new TSK 0
I20260619 20:49:43.523641 3500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:43.547695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35249
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:43.626523 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:43.626706 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:43.626729 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:43.627810 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:43.627846 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:43.627871 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:43.629225 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:43.629418 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:43.629575 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:43.634197 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:43.634835 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:43.634866 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:43.634887 3511 server_base.cc:1034] running on GCE node
I20260619 20:49:43.635138 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:43.635321 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:43.636485 3511 hybrid_clock.cc:648] HybridClock initialized: now 1781902183636468 us; error 33 us; skew 500 ppm
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:43.639091 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:43.639572 3511 webserver.cc:460] Webserver started at http://127.0.98.1:41203/ using document root <none> and password file <none>
I20260619 20:49:43.639808 3511 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:43.639876 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:43.640023 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:43.641055 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b237b40b1e24b0abee900abfda801b1"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "e1d7d39e70a0009e9df12618c2444ccf"
server_key_iv: "126de06c42d3c7b0ff5de42b33aaf686"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.641444 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b237b40b1e24b0abee900abfda801b1"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "e1d7d39e70a0009e9df12618c2444ccf"
server_key_iv: "126de06c42d3c7b0ff5de42b33aaf686"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.642694 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:43.643410 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.643582 3511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:43.643692 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "6b237b40b1e24b0abee900abfda801b1"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "e1d7d39e70a0009e9df12618c2444ccf"
server_key_iv: "126de06c42d3c7b0ff5de42b33aaf686"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.643774 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:43.658237 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:43.659910 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:43.660065 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:43.660279 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:43.660565 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:43.660617 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.660663 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:43.660717 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.666579 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42413
I20260619 20:49:43.666620 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42413 every 8 connection(s)
I20260619 20:49:43.666927 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:43.672015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3511
I20260619 20:49:43.672096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:43.672308 392 external_mini_cluster.cc:1428] Setting key cbfdf9b45a8a2ab4b7db0c32e86e66e5
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:43.676095 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.667623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50283 (local address 127.0.98.62:35249)
0619 20:49:43.667670 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:43.667673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:43.667861 (+ 188us) server_negotiation.cc:407] Connection header received
0619 20:49:43.668376 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:43.668378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:43.668389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:43.668441 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:43.669453 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.669955 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.670975 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.671177 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.671250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.673410 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:43.673412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:43.673412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:43.673420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:43.674750 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.675229 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.675230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.675230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.675275 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.675527 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.675527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.675528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.675615 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:43.675661 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:43.675971 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 20:49:43.676033 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:49:43.676393 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.667584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35249 (local address 127.0.98.1:50283)
0619 20:49:43.667732 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:43.667775 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:43.668274 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:43.668504 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:43.668508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:43.669125 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:43.669301 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:43.669305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.670047 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.670048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:43.670881 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:43.670883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.671256 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.671257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:43.671389 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.671820 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:43.671828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:43.673279 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:43.674831 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:43.674837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:43.674840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:43.675140 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:43.675335 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:43.675335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:43.675336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:43.675465 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:43.675675 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:43.675681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:43.675846 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 20:49:43.675956 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:49:43.676118 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260619 20:49:43.676889 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35249
I20260619 20:49:43.677001 3640 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:43.677152 3640 heartbeater.cc:511] Master 127.0.98.62:35249 requested a full tablet report, sending...
I20260619 20:49:43.677724 3449 ts_manager.cc:194] Registered new tserver with Master: 6b237b40b1e24b0abee900abfda801b1 (127.0.98.1:42413)
I20260619 20:49:43.678333 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50283
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:43.696440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35249
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:43.775764 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:43.775952 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:43.775974 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:43.777114 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:43.777151 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:43.777175 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:43.778539 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:43.778738 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:43.778934 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:43.783778 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:43.783913 3645 server_base.cc:1034] running on GCE node
W20260619 20:49:43.783798 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:43.783785 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:43.784224 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:43.784472 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:43.785617 3645 hybrid_clock.cc:648] HybridClock initialized: now 1781902183785606 us; error 46 us; skew 500 ppm
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:43.787932 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:43.788394 3645 webserver.cc:460] Webserver started at http://127.0.98.2:44367/ using document root <none> and password file <none>
I20260619 20:49:43.788604 3645 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:43.788666 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:43.788810 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:43.789883 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d0d996026ee463aae7a2b589b2691ce"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "54eed56de36ebe25eeaae54d1c031717"
server_key_iv: "6ed7de1d95c5dfb60495106e7c05a0a4"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.790225 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d0d996026ee463aae7a2b589b2691ce"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "54eed56de36ebe25eeaae54d1c031717"
server_key_iv: "6ed7de1d95c5dfb60495106e7c05a0a4"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.791497 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:43.792164 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.792321 3645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:43.792428 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "8d0d996026ee463aae7a2b589b2691ce"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "54eed56de36ebe25eeaae54d1c031717"
server_key_iv: "6ed7de1d95c5dfb60495106e7c05a0a4"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.792500 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:43.806313 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:43.808166 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:43.808326 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:43.808561 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:43.808869 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:43.808923 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.808960 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:43.808988 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.814733 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41599
I20260619 20:49:43.814783 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41599 every 8 connection(s)
I20260619 20:49:43.815076 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:43.821187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3645
I20260619 20:49:43.821301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:43.821547 392 external_mini_cluster.cc:1428] Setting key 7ec4ff47c944940fc480cf6736293d3d
I20260619 20:49:43.823848 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.815703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60259 (local address 127.0.98.62:35249)
0619 20:49:43.815746 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:43.815748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:43.815989 (+ 241us) server_negotiation.cc:407] Connection header received
0619 20:49:43.816423 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:43.816424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:43.816433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:43.816483 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:43.817409 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.817877 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.818767 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.818947 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.819000 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.820839 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:43.820840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:43.820840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:43.820847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:43.822523 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.822909 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.822909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.822910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.822974 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.823190 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.823191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.823192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.823335 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:43.823402 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:43.823679 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 20:49:43.823757 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:43.824034 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.815689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35249 (local address 127.0.98.2:60259)
0619 20:49:43.815842 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:43.815879 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:43.816340 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:43.816551 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:43.816554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:43.817089 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:43.817280 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:43.817284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.817948 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.817948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:43.818701 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:43.818703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.819017 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.819018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:43.819107 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.819483 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:43.819491 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:43.820755 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:43.822606 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:43.822612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:43.822613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:43.822838 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:43.823037 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:43.823038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:43.823038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:43.823131 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:43.823395 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:43.823399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:43.823546 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 20:49:43.823651 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:49:43.823815 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260619 20:49:43.824497 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35249
I20260619 20:49:43.824610 3774 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:43.824815 3774 heartbeater.cc:511] Master 127.0.98.62:35249 requested a full tablet report, sending...
I20260619 20:49:43.825194 3449 ts_manager.cc:194] Registered new tserver with Master: 8d0d996026ee463aae7a2b589b2691ce (127.0.98.2:41599)
I20260619 20:49:43.825739 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60259
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:43.844091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35249
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:43.921766 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:43.921965 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:43.921988 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:43.923054 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:43.923086 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:43.923112 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:43.924409 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:43.924574 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:43.924736 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:43.929504 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:43.929502 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:43.929533 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:43.929765 3779 server_base.cc:1034] running on GCE node
I20260619 20:49:43.929957 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:43.930188 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:43.931346 3779 hybrid_clock.cc:648] HybridClock initialized: now 1781902183931334 us; error 24 us; skew 500 ppm
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:43.933768 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:43.934194 3779 webserver.cc:460] Webserver started at http://127.0.98.3:35347/ using document root <none> and password file <none>
I20260619 20:49:43.934365 3779 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:43.934403 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:43.934486 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:43.935420 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e2903a09e99485da2a337db4fa5b39c"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "9113c27836dba55aa2aa1232b2087536"
server_key_iv: "14327e54f55b361023733a9a9dce4dbe"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.935715 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e2903a09e99485da2a337db4fa5b39c"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "9113c27836dba55aa2aa1232b2087536"
server_key_iv: "14327e54f55b361023733a9a9dce4dbe"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.936985 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:49:43.937680 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.937865 3779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.937944 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "0e2903a09e99485da2a337db4fa5b39c"
format_stamp: "Formatted at 2026-06-19 20:49:43 on dist-test-slave-0smx"
server_key: "9113c27836dba55aa2aa1232b2087536"
server_key_iv: "14327e54f55b361023733a9a9dce4dbe"
server_key_version: "encryptionkey@0"
I20260619 20:49:43.938042 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:43.958814 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:43.961071 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:43.961176 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:43.961404 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:43.961716 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:43.961771 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.961812 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:43.961853 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:43.967631 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45465
I20260619 20:49:43.967698 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45465 every 8 connection(s)
I20260619 20:49:43.968017 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:43.968518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3779
I20260619 20:49:43.968621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:43.968844 392 external_mini_cluster.cc:1428] Setting key bb39e8521cf18f708880381898225f1c
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:43.976310 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.968679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40103 (local address 127.0.98.62:35249)
0619 20:49:43.968724 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:43.968727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:43.968956 (+ 229us) server_negotiation.cc:407] Connection header received
0619 20:49:43.969629 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:43.969629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:43.969640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:43.969693 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:43.970585 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.971069 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.971975 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.972152 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.972200 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.974099 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:43.974100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:43.974100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:43.974105 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:43.975269 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.975600 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.975600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.975601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.975648 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:43.975850 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:43.975851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:43.975851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:43.975939 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:43.975985 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:43.976192 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 20:49:43.976247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:49:43.976475 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.968687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35249 (local address 127.0.98.3:40103)
0619 20:49:43.968840 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:43.968878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:43.969529 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:43.969779 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:43.969783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:43.970334 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:43.970512 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:43.970516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.971141 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.971142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:43.971909 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:43.971911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.972217 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.972218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:43.972307 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.972679 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:43.972686 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:43.974019 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:43.975334 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:43.975338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:43.975340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:43.975541 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:43.975702 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:43.975702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:43.975703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:43.975799 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:43.975991 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:43.975994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:43.976092 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 20:49:43.976168 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:49:43.976272 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260619 20:49:43.976910 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35249
I20260619 20:49:43.977005 3908 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:43.977175 3908 heartbeater.cc:511] Master 127.0.98.62:35249 requested a full tablet report, sending...
I20260619 20:49:43.977574 3449 ts_manager.cc:194] Registered new tserver with Master: 0e2903a09e99485da2a337db4fa5b39c (127.0.98.3:45465)
I20260619 20:49:43.977912 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40103
I20260619 20:49:43.980537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 20:49:43 dist-test-slave-0smx krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902183, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:44.002213 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:43.995390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.98.62:35249)
0619 20:49:43.995428 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:43.995430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:43.995549 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:49:43.995604 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:43.995605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:43.995613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:43.995673 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:43.996190 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.996865 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.997639 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:43.997819 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:43.997873 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:43.999965 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:43.999966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:43.999967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:43.999973 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:44.001206 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:44.001565 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:44.001566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:44.001566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:44.001616 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:44.001782 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:44.001782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:44.001782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:44.001910 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:44.001984 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:44.002064 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 20:49:44.002121 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:49:44.002462 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51974
I20260619 20:49:44.003082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3511
I20260619 20:49:44.007454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3645
I20260619 20:49:44.011150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3779
I20260619 20:49:44.015170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3417
2026-06-19T20:49:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1793 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:44 dist-test-slave-0smx krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:44 dist-test-slave-0smx krb5kdc[3920](info): set up 2 sockets
Jun 19 20:49:44 dist-test-slave-0smx krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260619 20:49:45.053191 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:45.167915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:45.246273 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:45.246531 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:45.246568 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:45.247781 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:45.247855 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:45.247881 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:45.247915 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:45.247962 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:45.249488 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:45.249768 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:45.250006 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:45.254711 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.254915 3933 server_base.cc:1034] running on GCE node
W20260619 20:49:45.254765 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:45.254709 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.255239 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:45.255538 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:45.256703 3933 hybrid_clock.cc:648] HybridClock initialized: now 1781902185256688 us; error 26 us; skew 500 ppm
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:45.259166 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:45.259553 3933 webserver.cc:460] Webserver started at http://127.0.98.62:35699/ using document root <none> and password file <none>
I20260619 20:49:45.259791 3933 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:45.259838 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:45.260025 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:45.261126 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "7be354a403d84516b70ebc2b7bc08b09"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "3b4deaa641a696b2ef427da211c1a6db"
server_key_iv: "e01ec4e2fe218d5d70704bf0704fc599"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.261548 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "7be354a403d84516b70ebc2b7bc08b09"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "3b4deaa641a696b2ef427da211c1a6db"
server_key_iv: "e01ec4e2fe218d5d70704bf0704fc599"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.262975 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:45.263757 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.263931 3933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:45.264041 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7be354a403d84516b70ebc2b7bc08b09"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "3b4deaa641a696b2ef427da211c1a6db"
server_key_iv: "e01ec4e2fe218d5d70704bf0704fc599"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.264122 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:45.279532 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:45.281565 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:45.281733 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:45.285514 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44741
I20260619 20:49:45.285568 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44741 every 8 connection(s)
I20260619 20:49:45.285939 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:45.286754 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:45.288947 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09: Bootstrap starting.
I20260619 20:49:45.289554 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:45.289799 4001 log.cc:824] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:45.290390 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09: No bootstrap required, opened a new log
I20260619 20:49:45.291901 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be354a403d84516b70ebc2b7bc08b09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44741 } }
I20260619 20:49:45.292129 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:45.292150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 3933
I20260619 20:49:45.292183 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be354a403d84516b70ebc2b7bc08b09, State: Initialized, Role: FOLLOWER
I20260619 20:49:45.292235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:45.292342 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be354a403d84516b70ebc2b7bc08b09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44741 } }
I20260619 20:49:45.292407 392 external_mini_cluster.cc:1428] Setting key 1167c08c6b8cbc98c56857883beb8cf1
I20260619 20:49:45.292423 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:45.292459 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:45.292506 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:45.293547 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be354a403d84516b70ebc2b7bc08b09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44741 } }
I20260619 20:49:45.293660 4001 leader_election.cc:302] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7be354a403d84516b70ebc2b7bc08b09; no voters:
I20260619 20:49:45.293820 4001 leader_election.cc:288] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:45.293874 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:45.294024 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [term 1 LEADER]: Becoming Leader. State: Replica: 7be354a403d84516b70ebc2b7bc08b09, State: Running, Role: LEADER
I20260619 20:49:45.294075 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:45.294188 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be354a403d84516b70ebc2b7bc08b09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44741 } }
I20260619 20:49:45.294567 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be354a403d84516b70ebc2b7bc08b09. Latest consensus state: current_term: 1 leader_uuid: "7be354a403d84516b70ebc2b7bc08b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be354a403d84516b70ebc2b7bc08b09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44741 } } }
I20260619 20:49:45.294586 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7be354a403d84516b70ebc2b7bc08b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be354a403d84516b70ebc2b7bc08b09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44741 } } }
I20260619 20:49:45.294716 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:45.294726 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:45.294929 4015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:45.295418 4015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:45.297026 4015 catalog_manager.cc:1347] Generated new cluster ID: f83edc143f4f485bb10a098d18d40379
I20260619 20:49:45.297084 4015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:45.303880 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.292871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.0.98.62:44741)
0619 20:49:45.293047 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:45.293062 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:45.293086 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:49:45.293171 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:45.293178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:45.293217 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:45.293388 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:45.294154 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.295745 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.296761 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.297350 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.297437 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.300083 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:45.300086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:45.300087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:45.300098 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:45.302072 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.302446 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.302449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.302449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.302788 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.302991 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.302992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.302992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.303298 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:45.303455 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:45.303524 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 20:49:45.303654 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260619 20:49:45.319000 4015 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:45.319505 4015 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:45.328235 4015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7be354a403d84516b70ebc2b7bc08b09: Generated new TSK 0
I20260619 20:49:45.328440 4015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:45.355573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44741
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:45.434978 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:45.435274 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:45.435313 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:45.437228 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:45.437342 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:45.437393 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:45.438900 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:45.439072 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:45.439241 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:45.444120 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:45.444126 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.444315 4027 server_base.cc:1034] running on GCE node
W20260619 20:49:45.444185 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.444548 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:45.444751 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:45.445931 4027 hybrid_clock.cc:648] HybridClock initialized: now 1781902185445935 us; error 43 us; skew 500 ppm
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:45.448331 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:45.448740 4027 webserver.cc:460] Webserver started at http://127.0.98.1:42019/ using document root <none> and password file <none>
I20260619 20:49:45.448951 4027 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:45.448997 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:45.449157 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:45.450233 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "0e793c09f8114de9b82ece7ed0a567b3"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "da350508c304e850a995d6b57b148b13"
server_key_iv: "16c919d1567b98f4fecd9f953984690b"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.450580 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0e793c09f8114de9b82ece7ed0a567b3"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "da350508c304e850a995d6b57b148b13"
server_key_iv: "16c919d1567b98f4fecd9f953984690b"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.451987 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:45.452718 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.452907 4027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:45.452986 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "0e793c09f8114de9b82ece7ed0a567b3"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "da350508c304e850a995d6b57b148b13"
server_key_iv: "16c919d1567b98f4fecd9f953984690b"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.453039 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:45.468550 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:45.470376 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:45.470480 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:45.470669 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:45.470904 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:45.470932 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.470953 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:45.470966 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.476213 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43161
I20260619 20:49:45.476281 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43161 every 8 connection(s)
I20260619 20:49:45.476599 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:45.480145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4027
I20260619 20:49:45.480259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:45.480443 392 external_mini_cluster.cc:1428] Setting key f01f2f22e92ec27a83bffc9f513ea139
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:45.486645 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.477271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44741 (local address 127.0.98.1:39015)
0619 20:49:45.477449 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:45.477482 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:45.478341 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:45.478540 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:45.478543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:45.478779 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:45.478980 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:45.478983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.479594 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.479595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:45.480725 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:45.480727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.481210 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.481211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:45.481374 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.481879 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:45.481888 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:45.483614 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:45.485427 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:45.485432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:45.485434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:45.485632 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:45.485843 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:45.485843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:45.485843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:45.485930 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:45.486215 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:45.486218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:45.486320 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 20:49:45.486397 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:49:45.486501 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260619 20:49:45.486670 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.477285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39015 (local address 127.0.98.62:44741)
0619 20:49:45.477355 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:45.477357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:45.477542 (+ 185us) server_negotiation.cc:407] Connection header received
0619 20:49:45.478428 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:45.478429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:45.478438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:45.478483 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:45.479046 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.479516 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.480805 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.481121 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.481193 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.483706 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:45.483707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:45.483708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:45.483717 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:45.485355 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.485708 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.485709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.485710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.485773 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.485994 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.485996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.485996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.486150 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:45.486224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:45.486457 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 20:49:45.486543 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:49:45.487202 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44741
I20260619 20:49:45.487306 4156 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:45.487488 4156 heartbeater.cc:511] Master 127.0.98.62:44741 requested a full tablet report, sending...
I20260619 20:49:45.488000 3965 ts_manager.cc:194] Registered new tserver with Master: 0e793c09f8114de9b82ece7ed0a567b3 (127.0.98.1:43161)
I20260619 20:49:45.488588 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39015
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:45.505990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44741
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:45.583906 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:45.584141 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:45.584180 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:45.585323 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:45.585386 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:45.585429 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:45.586885 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:45.587139 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:45.587363 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:45.592278 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:45.592284 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:45.592296 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.592555 4161 server_base.cc:1034] running on GCE node
I20260619 20:49:45.592747 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:45.592980 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:45.594132 4161 hybrid_clock.cc:648] HybridClock initialized: now 1781902185594124 us; error 29 us; skew 500 ppm
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:45.596668 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:45.597090 4161 webserver.cc:460] Webserver started at http://127.0.98.2:32983/ using document root <none> and password file <none>
I20260619 20:49:45.597378 4161 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:45.597445 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:45.597595 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:45.598580 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "b614ca9b5e66460eaff7ccaec1f33960"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "393fa1bceba7d78f2c27dd0dd281a39a"
server_key_iv: "a5d5a803c97b4f41785f06394387dfff"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.598942 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b614ca9b5e66460eaff7ccaec1f33960"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "393fa1bceba7d78f2c27dd0dd281a39a"
server_key_iv: "a5d5a803c97b4f41785f06394387dfff"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.600234 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:45.600921 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.601086 4161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:45.601187 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "b614ca9b5e66460eaff7ccaec1f33960"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "393fa1bceba7d78f2c27dd0dd281a39a"
server_key_iv: "a5d5a803c97b4f41785f06394387dfff"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.601267 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:45.607422 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:45.609575 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:45.609716 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:45.609917 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:45.610239 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:45.610288 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.610327 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:45.610353 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.615969 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33423
I20260619 20:49:45.616030 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33423 every 8 connection(s)
I20260619 20:49:45.616330 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:45.620385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4161
I20260619 20:49:45.620492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:45.620656 392 external_mini_cluster.cc:1428] Setting key 13158b96c18dfda5060df727f8ab89b0
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:45.626361 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.617069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60851 (local address 127.0.98.62:44741)
0619 20:49:45.617134 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:45.617136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:45.617328 (+ 192us) server_negotiation.cc:407] Connection header received
0619 20:49:45.618250 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:45.618251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:45.618262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:45.618323 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:45.618914 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.619412 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.620421 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.620676 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.620788 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.623341 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:45.623342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:45.623342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:45.623367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:45.624884 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.625418 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.625419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.625419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.625469 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.625765 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.625765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.625766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.625859 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:45.625904 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:45.626242 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 20:49:45.626301 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:49:45.626680 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.617048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44741 (local address 127.0.98.2:60851)
0619 20:49:45.617201 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:45.617232 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:45.618144 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:45.618393 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:45.618396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:45.618646 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:45.618815 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:45.618819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.619502 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.619503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:45.620309 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:45.620311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.620796 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.620797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:45.620918 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.621411 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:45.621418 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:45.623238 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:45.624989 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:45.624995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:45.624997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:45.625339 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:45.625553 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:45.625553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:45.625554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:45.625695 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:45.625920 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:45.625926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:45.626100 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 20:49:45.626212 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:49:45.626385 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260619 20:49:45.627192 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44741
I20260619 20:49:45.627290 4290 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:45.627472 4290 heartbeater.cc:511] Master 127.0.98.62:44741 requested a full tablet report, sending...
I20260619 20:49:45.627795 3965 ts_manager.cc:194] Registered new tserver with Master: b614ca9b5e66460eaff7ccaec1f33960 (127.0.98.2:33423)
I20260619 20:49:45.628189 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60851
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:45.645740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44741
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:45.723510 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:45.723724 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:45.723759 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:45.724848 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:45.724911 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:45.724954 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:45.726383 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:45.726630 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:45.726897 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:45.731657 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:45.731653 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.731729 4295 server_base.cc:1034] running on GCE node
W20260619 20:49:45.731660 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:45.732057 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:45.732299 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:45.733431 4295 hybrid_clock.cc:648] HybridClock initialized: now 1781902185733424 us; error 34 us; skew 500 ppm
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:45.735755 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:45.736186 4295 webserver.cc:460] Webserver started at http://127.0.98.3:41245/ using document root <none> and password file <none>
I20260619 20:49:45.736423 4295 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:45.736467 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:45.736605 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:45.737658 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "e396b22a1de54248b2c1a7b41952b0eb"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "8cfd4e4b58485a6c15c20589a1206a9c"
server_key_iv: "256c6691f49df31709b489ef2f6017fe"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.738000 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e396b22a1de54248b2c1a7b41952b0eb"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "8cfd4e4b58485a6c15c20589a1206a9c"
server_key_iv: "256c6691f49df31709b489ef2f6017fe"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.739230 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:45.739885 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.740090 4295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:45.740154 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "e396b22a1de54248b2c1a7b41952b0eb"
format_stamp: "Formatted at 2026-06-19 20:49:45 on dist-test-slave-0smx"
server_key: "8cfd4e4b58485a6c15c20589a1206a9c"
server_key_iv: "256c6691f49df31709b489ef2f6017fe"
server_key_version: "encryptionkey@0"
I20260619 20:49:45.740243 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:45.758261 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:45.760066 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:45.760215 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:45.760456 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:45.760771 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:45.760825 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.760860 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:45.760887 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:45.766559 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34327
I20260619 20:49:45.766610 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34327 every 8 connection(s)
I20260619 20:49:45.766938 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:45.770097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4295
I20260619 20:49:45.770206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:45.770411 392 external_mini_cluster.cc:1428] Setting key a6d76461726270463fe82fa38b0a40b6
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:45.775767 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.767581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56095 (local address 127.0.98.62:44741)
0619 20:49:45.767630 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:45.767632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:45.767859 (+ 227us) server_negotiation.cc:407] Connection header received
0619 20:49:45.768802 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:45.768803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:45.768815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:45.768856 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:45.769533 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.770125 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.771255 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.771445 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.771497 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.773448 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:45.773449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:45.773450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:45.773455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:45.774750 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.775077 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.775077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.775078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.775122 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.775317 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.775318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.775318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.775407 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:45.775453 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:45.775640 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 20:49:45.775691 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:49:45.775933 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.767588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44741 (local address 127.0.98.3:56095)
0619 20:49:45.767750 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:45.767782 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:45.768690 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:45.768930 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:45.768934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:45.769190 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:45.769444 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:45.769449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.770339 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.770340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:45.771192 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:45.771194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.771519 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.771521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:45.771613 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.772009 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:45.772035 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:45.773374 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:45.774820 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:45.774823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:45.774825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:45.775010 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:45.775183 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:45.775184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:45.775184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:45.775267 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:45.775458 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:45.775461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:45.775556 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 20:49:45.775629 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:49:45.775729 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260619 20:49:45.776400 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44741
I20260619 20:49:45.776485 4424 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:45.776655 4424 heartbeater.cc:511] Master 127.0.98.62:44741 requested a full tablet report, sending...
I20260619 20:49:45.776978 3965 ts_manager.cc:194] Registered new tserver with Master: e396b22a1de54248b2c1a7b41952b0eb (127.0.98.3:34327)
I20260619 20:49:45.777338 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56095
I20260619 20:49:45.781874 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:45.803411 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.796306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.0.98.1:43161)
0619 20:49:45.796455 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:45.796459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:45.796469 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:45.796492 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:45.796494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:45.796506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:45.796566 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:45.797097 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.797850 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.798548 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.798853 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.798905 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.800917 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:45.800923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:45.800924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:45.800930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:45.802311 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.802659 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.802661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.802662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.802712 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.802923 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.802924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.802924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.803040 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:45.803148 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:45.803234 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 20:49:45.803291 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:45.810850 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:45.804561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.0.98.62:44741)
0619 20:49:45.804601 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:45.804603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:45.804616 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:49:45.804664 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:45.804665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:45.804672 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:45.804713 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:45.805156 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.805631 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.806261 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:45.806425 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:45.806472 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:45.808473 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:45.808474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:45.808474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:45.808480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:45.809926 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.810247 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.810248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.810248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.810293 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:45.810456 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:45.810456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:45.810457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:45.810579 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:45.810650 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:45.810740 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:49:45.810791 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:49:45.811092 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44446
I20260619 20:49:45.811458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4027
I20260619 20:49:45.816350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4161
I20260619 20:49:45.820335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4295
I20260619 20:49:45.824445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 3933
2026-06-19T20:49:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1809 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[4436](info): set up 2 sockets
Jun 19 20:49:45 dist-test-slave-0smx krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260619 20:49:46.869179 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:46 dist-test-slave-0smx krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:46.980021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:47.059016 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:47.059213 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:47.059235 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:47.060283 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:47.060316 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:47.060330 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:47.060343 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:47.060354 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:47.061733 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:47.061904 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:47.062063 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:49:47.066759 4449 server_base.cc:1034] running on GCE node
W20260619 20:49:47.066735 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.066741 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.066991 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:47.067251 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:47.067559 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:47.068717 4449 hybrid_clock.cc:648] HybridClock initialized: now 1781902187068700 us; error 29 us; skew 500 ppm
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:47.071141 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:47.071462 4449 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:47.071532 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:47.071673 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:47.072667 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "be881e29da704f60b03cf7f58e4b8318"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "a95d1d93b044c7bdb26ad91c767f7902"
server_key_iv: "68ea8398a269594b4b971e191ee0b64a"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.073011 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "be881e29da704f60b03cf7f58e4b8318"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "a95d1d93b044c7bdb26ad91c767f7902"
server_key_iv: "68ea8398a269594b4b971e191ee0b64a"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.074338 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:49:47.075011 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.075174 4449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.075269 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "be881e29da704f60b03cf7f58e4b8318"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "a95d1d93b044c7bdb26ad91c767f7902"
server_key_iv: "68ea8398a269594b4b971e191ee0b64a"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.075349 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:47.100875 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:47.102622 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:47.102775 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:47.106478 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42185
I20260619 20:49:47.106509 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42185 every 8 connection(s)
I20260619 20:49:47.106782 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:47.107704 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:47.109903 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: Bootstrap starting.
I20260619 20:49:47.110414 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:47.110643 4516 log.cc:824] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:47.111254 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: No bootstrap required, opened a new log
I20260619 20:49:47.112704 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be881e29da704f60b03cf7f58e4b8318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 20:49:47.112833 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:47.112854 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be881e29da704f60b03cf7f58e4b8318, State: Initialized, Role: FOLLOWER
I20260619 20:49:47.112983 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be881e29da704f60b03cf7f58e4b8318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 20:49:47.113067 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:47.113090 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:47.113117 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:47.113845 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be881e29da704f60b03cf7f58e4b8318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 20:49:47.113924 4516 leader_election.cc:302] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be881e29da704f60b03cf7f58e4b8318; no voters:
I20260619 20:49:47.114049 4516 leader_election.cc:288] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:47.114117 4519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:47.114279 4519 raft_consensus.cc:686] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [term 1 LEADER]: Becoming Leader. State: Replica: be881e29da704f60b03cf7f58e4b8318, State: Running, Role: LEADER
I20260619 20:49:47.114307 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:47.114382 4519 consensus_queue.cc:237] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be881e29da704f60b03cf7f58e4b8318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 20:49:47.114595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4449
I20260619 20:49:47.114718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:47.114959 392 external_mini_cluster.cc:1428] Setting key 837737b99a6eed979840f3365c555328
I20260619 20:49:47.115541 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be881e29da704f60b03cf7f58e4b8318. Latest consensus state: current_term: 1 leader_uuid: "be881e29da704f60b03cf7f58e4b8318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be881e29da704f60b03cf7f58e4b8318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } }
I20260619 20:49:47.115550 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be881e29da704f60b03cf7f58e4b8318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be881e29da704f60b03cf7f58e4b8318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } }
I20260619 20:49:47.115644 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:47.115654 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318 [sys.catalog]: This master's current role is: LEADER
W20260619 20:49:47.116510 4537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:49:47.116606 4537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:49:47.116732 4533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:47.116917 4533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:47.118412 4533 catalog_manager.cc:1347] Generated new cluster ID: 0020d7ba68524fc4914df623f9bd6fc2
I20260619 20:49:47.118464 4533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:47.126075 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.115540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.0.98.62:42185)
0619 20:49:47.115700 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.115710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.115745 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:49:47.115867 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.115876 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.115934 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.116765 (+ 831us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:47.116874 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.118057 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.119137 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.119776 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.119891 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.122615 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:47.122619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:47.122619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:47.122631 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:47.124486 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.124886 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.124888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.124889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.124984 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.125189 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.125190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.125190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.125479 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:47.125619 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.125752 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.125877 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260619 20:49:47.129640 4533 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:47.130124 4533 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:47.134061 4533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: Generated new TSK 0
I20260619 20:49:47.134229 4533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:47.161779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42185
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:47.241346 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:47.241559 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:47.241583 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:47.242753 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:47.242794 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:47.242821 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:47.244313 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:47.244551 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:47.244798 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:47.249581 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.249581 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.249622 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:47.249740 4542 server_base.cc:1034] running on GCE node
I20260619 20:49:47.250031 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:47.250303 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:47.251464 4542 hybrid_clock.cc:648] HybridClock initialized: now 1781902187251459 us; error 35 us; skew 500 ppm
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:47.256788 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:47.257135 4542 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:47.257211 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:47.257382 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:47.258442 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "260716538087475d85549c9721ba881b"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "23f6b13384e93c0976db4e16d28a076b"
server_key_iv: "75897d580a775192cb0de5a43f21243c"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.258831 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "260716538087475d85549c9721ba881b"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "23f6b13384e93c0976db4e16d28a076b"
server_key_iv: "75897d580a775192cb0de5a43f21243c"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.260084 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:47.260783 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.260977 4542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:47.261089 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "260716538087475d85549c9721ba881b"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "23f6b13384e93c0976db4e16d28a076b"
server_key_iv: "75897d580a775192cb0de5a43f21243c"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.261166 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:47.269923 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:47.271631 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:47.271786 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:47.272010 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:47.272318 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:47.272382 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.272425 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:47.272449 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.277999 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46351
I20260619 20:49:47.278046 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46351 every 8 connection(s)
I20260619 20:49:47.278326 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:47.286162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4542
I20260619 20:49:47.286271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:47.286526 392 external_mini_cluster.cc:1428] Setting key 09dc9b19aec316235cf1643cf8a02d41
I20260619 20:49:47.287205 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.279137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40615 (local address 127.0.98.62:42185)
0619 20:49:47.279214 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.279216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.279381 (+ 165us) server_negotiation.cc:407] Connection header received
0619 20:49:47.279816 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.279817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.279831 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.279882 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:47.280740 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.281230 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.282262 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.282450 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.282501 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.284369 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:47.284370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:47.284371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:47.284377 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:47.285779 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.286123 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.286124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.286124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.286188 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.286464 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.286465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.286466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.286616 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:47.286696 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.287022 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.287087 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:49:47.287408 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.279105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.98.1:40615)
0619 20:49:47.279258 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:47.279292 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:47.279736 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:47.279934 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:47.279937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:47.280174 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:47.280660 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.280664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.281388 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.281389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.282179 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.282181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.282517 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.282518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.282612 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.283030 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:47.283037 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:47.284280 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:47.285843 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:47.285848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:47.285850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:47.286044 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:47.286262 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:47.286263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:47.286263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:47.286399 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:47.286686 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:47.286705 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:47.286888 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 20:49:47.286997 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 20:49:47.287147 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260619 20:49:47.287890 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42185
I20260619 20:49:47.288017 4670 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:47.288165 4670 heartbeater.cc:511] Master 127.0.98.62:42185 requested a full tablet report, sending...
I20260619 20:49:47.288654 4480 ts_manager.cc:194] Registered new tserver with Master: 260716538087475d85549c9721ba881b (127.0.98.1:46351)
I20260619 20:49:47.289597 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40615
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:47.311560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42185
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:47.390910 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:47.391109 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:47.391134 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:47.392196 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:47.392227 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:47.392253 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:47.393641 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:47.393803 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:47.393965 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:47.398778 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.398787 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:47.398834 4675 server_base.cc:1034] running on GCE node
W20260619 20:49:47.398787 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:47.399230 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:47.399482 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:47.400626 4675 hybrid_clock.cc:648] HybridClock initialized: now 1781902187400615 us; error 28 us; skew 500 ppm
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:47.403065 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:47.403375 4675 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:47.403447 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:47.403595 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:47.404582 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "a38a58c7b98f4ebf8c5c3fde1b352028"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "e7c490d3ba2cb0c1f29a32ac590f08bb"
server_key_iv: "b223dad559b4d6b66260fd021a648765"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.404942 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a38a58c7b98f4ebf8c5c3fde1b352028"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "e7c490d3ba2cb0c1f29a32ac590f08bb"
server_key_iv: "b223dad559b4d6b66260fd021a648765"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.406281 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:47.406980 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.407151 4675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:47.407245 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "a38a58c7b98f4ebf8c5c3fde1b352028"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "e7c490d3ba2cb0c1f29a32ac590f08bb"
server_key_iv: "b223dad559b4d6b66260fd021a648765"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.407322 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:47.420374 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:47.422519 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:47.422675 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:47.422932 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:47.423269 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:47.423311 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.423346 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:47.423377 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.430577 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39511
I20260619 20:49:47.430604 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39511 every 8 connection(s)
I20260619 20:49:47.430927 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:47.436084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4675
I20260619 20:49:47.436200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:47.436393 392 external_mini_cluster.cc:1428] Setting key cdeebaf990069aebd8b0188673252291
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:47.441239 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.431677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45533 (local address 127.0.98.62:42185)
0619 20:49:47.431732 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.431734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.432014 (+ 280us) server_negotiation.cc:407] Connection header received
0619 20:49:47.432512 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.432514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.432527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.432602 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:47.433534 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.434102 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.435093 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.435280 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.435339 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.437965 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:47.437966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:47.437967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:47.437974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:47.439549 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.440122 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.440123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.440123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.440170 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.440505 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.440505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.440505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.440603 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:47.440649 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.441079 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.441162 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:49:47.441644 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.431644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.98.2:45533)
0619 20:49:47.431880 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:47.431910 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:47.432417 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:47.432653 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:47.432656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:47.432912 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:47.433452 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.433457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.434183 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.434184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.435009 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.435011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.435355 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.435356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.435449 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.435942 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:47.435952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:47.437870 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:47.439639 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:47.439647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:47.439657 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:47.440034 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:47.440247 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:47.440249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:47.440249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:47.440425 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:47.440679 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:47.440686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:47.440891 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 20:49:47.441034 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 20:49:47.441250 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260619 20:49:47.442330 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42185
I20260619 20:49:47.442443 4803 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:47.442610 4803 heartbeater.cc:511] Master 127.0.98.62:42185 requested a full tablet report, sending...
I20260619 20:49:47.442989 4480 ts_manager.cc:194] Registered new tserver with Master: a38a58c7b98f4ebf8c5c3fde1b352028 (127.0.98.2:39511)
I20260619 20:49:47.443427 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45533
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 20:49:47.547829 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.111s user 0.000s sys 0.003s
I20260619 20:49:47.548187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42185
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:47.628038 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:47.628261 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:47.628300 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:47.629531 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:47.629592 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:47.629639 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:47.631110 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:47.631366 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:47.631580 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:47.636647 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.636638 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:47.636608 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:47.636828 4808 server_base.cc:1034] running on GCE node
I20260619 20:49:47.637065 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:47.637272 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:47.638484 4808 hybrid_clock.cc:648] HybridClock initialized: now 1781902187638472 us; error 28 us; skew 500 ppm
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:47.640869 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:47.641144 4808 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:47.641186 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:47.641270 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:47.642297 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "a6954f8f2fa84a02acb153e68f6d2904"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "28dbb349d51cf3a4fbefab28c3ee2186"
server_key_iv: "1b0a226833e3bdca4392b1cf45d48327"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.642606 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a6954f8f2fa84a02acb153e68f6d2904"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "28dbb349d51cf3a4fbefab28c3ee2186"
server_key_iv: "1b0a226833e3bdca4392b1cf45d48327"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.643838 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:49:47.644510 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.644680 4808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:47.644765 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "a6954f8f2fa84a02acb153e68f6d2904"
format_stamp: "Formatted at 2026-06-19 20:49:47 on dist-test-slave-0smx"
server_key: "28dbb349d51cf3a4fbefab28c3ee2186"
server_key_iv: "1b0a226833e3bdca4392b1cf45d48327"
server_key_version: "encryptionkey@0"
I20260619 20:49:47.644817 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:47.663439 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:47.665606 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:47.665750 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:47.665983 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:47.666294 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:47.666326 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.666380 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:47.666409 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:47.671780 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39229
I20260619 20:49:47.671842 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39229 every 8 connection(s)
I20260619 20:49:47.672097 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:47.672550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4808
I20260619 20:49:47.672636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:47.672837 392 external_mini_cluster.cc:1428] Setting key 02f19963ff36d98ed1c58102e9c40bac
Jun 19 20:49:47 dist-test-slave-0smx krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:47.681197 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.672884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57245 (local address 127.0.98.62:42185)
0619 20:49:47.672932 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.672934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.673192 (+ 258us) server_negotiation.cc:407] Connection header received
0619 20:49:47.673781 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.673782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.673793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.673849 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:47.674754 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.675256 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.676223 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.676426 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.676513 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.678606 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:47.678608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:47.678608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:47.678615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:47.679846 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.680251 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.680252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.680253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.680308 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.680555 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.680556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.680556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.680672 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:47.680757 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.681032 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.681106 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:49:47.681392 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.672884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.98.3:57245)
0619 20:49:47.673076 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:47.673121 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:47.673672 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:47.673915 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:47.673919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:47.674191 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:47.674657 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.674662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.675346 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.675348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.676121 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.676122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.676521 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.676522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.676640 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.677025 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:47.677032 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:47.678510 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:47.679940 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:47.679945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:47.679947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:47.680160 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:47.680376 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:47.680377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:47.680377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:47.680486 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:47.680739 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:47.680753 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:47.680886 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:49:47.681007 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 20:49:47.681152 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260619 20:49:47.681836 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42185
I20260619 20:49:47.681951 4936 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:47.682101 4936 heartbeater.cc:511] Master 127.0.98.62:42185 requested a full tablet report, sending...
I20260619 20:49:47.682435 4480 ts_manager.cc:194] Registered new tserver with Master: a6954f8f2fa84a02acb153e68f6d2904 (127.0.98.3:39229)
I20260619 20:49:47.682809 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57245
I20260619 20:49:47.684475 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:49:47.691386 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.685714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.0.98.62:42185)
0619 20:49:47.685778 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.685780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.685885 (+ 105us) server_negotiation.cc:407] Connection header received
0619 20:49:47.685961 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.685961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.685971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.686021 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:47.686577 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.687047 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.687774 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.687978 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.688061 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.689096 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:47.689097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:47.689098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:47.689105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:47.690249 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.690599 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.690600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.690600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.690651 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:47.690870 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:47.690871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:47.690871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:47.691009 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:47.691116 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.691223 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.691293 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:49:47.692740 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:47.694288 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:47.700264 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.696455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46351 (local address 127.0.0.1:36542)
0619 20:49:47.696660 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:47.696698 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:47.696766 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:47.697091 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:47.697092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:47.697095 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:47.697146 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.697147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.698087 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.698088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.699139 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.699141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.699968 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.699976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.700052 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.700064 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:49:47.700148 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:49:47.700198 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260619 20:49:47.700284 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.696565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.98.1:46351)
0619 20:49:47.696898 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.696902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.696912 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:47.696947 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.696949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.696963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.697079 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:47.697204 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.697996 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.699243 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.699886 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.699969 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.700027 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.700174 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.700209 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":92,"threads_started":1}
I20260619 20:49:47.700670 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.696633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39229 (local address 127.0.0.1:40988)
0619 20:49:47.697086 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:47.697095 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:47.697140 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:47.697348 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:47.697349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:47.697351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:47.697382 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.697383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.698620 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.698621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.699477 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.699479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.700426 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.700427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.700485 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.700490 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:47.700546 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:49:47.700581 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":61,"threads_started":1}
I20260619 20:49:47.700855 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.696865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.0.98.3:39229)
0619 20:49:47.697040 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.697044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.697190 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:49:47.697227 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.697229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.697243 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.697470 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:47.697544 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.698548 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.699555 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.700259 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.700329 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.700391 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.700739 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.700781 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260619 20:49:47.701121 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 36000d82623243c8b1d7ed89e5f46f11 (DEFAULT_TABLE table=test-table [id=76a32e5fe48c46adb61680d37037256d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:47.701507 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36000d82623243c8b1d7ed89e5f46f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:47.701452 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 36000d82623243c8b1d7ed89e5f46f11 (DEFAULT_TABLE table=test-table [id=76a32e5fe48c46adb61680d37037256d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:47.701778 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36000d82623243c8b1d7ed89e5f46f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:47.703899 4953 tablet_bootstrap.cc:492] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: Bootstrap starting.
I20260619 20:49:47.704175 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.696650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.98.2:39511)
0619 20:49:47.696815 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:47.696820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:47.696832 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:49:47.696871 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:47.696873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:47.696889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:47.696968 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:47.697164 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.698450 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.699681 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.700361 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.700687 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.700951 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:47.704040 (+ 3089us) server_negotiation.cc:299] Negotiation successful
0619 20:49:47.704082 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260619 20:49:47.704360 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:47.696455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39511 (local address 127.0.0.1:58988)
0619 20:49:47.696740 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:47.696749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:47.696797 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:47.696998 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:47.697002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:47.697005 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:47.697068 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.697069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.698566 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.698567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.699605 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:47.699607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:47.700693 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:47.700694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:47.703928 (+ 3234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:47.703933 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:49:47.704249 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 20:49:47.704295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":4,"thread_start_us":53,"threads_started":1}
I20260619 20:49:47.704661 4953 tablet_bootstrap.cc:654] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:47.705032 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 36000d82623243c8b1d7ed89e5f46f11 (DEFAULT_TABLE table=test-table [id=76a32e5fe48c46adb61680d37037256d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:47.705264 4953 log.cc:824] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:47.705397 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36000d82623243c8b1d7ed89e5f46f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:47.705451 4954 tablet_bootstrap.cc:492] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Bootstrap starting.
I20260619 20:49:47.706069 4954 tablet_bootstrap.cc:654] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:47.706097 4953 tablet_bootstrap.cc:492] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: No bootstrap required, opened a new log
I20260619 20:49:47.706156 4953 ts_tablet_manager.cc:1397] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:49:47.706573 4954 log.cc:824] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:47.708199 4953 raft_consensus.cc:348] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:47.708365 4953 raft_consensus.cc:374] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:47.708401 4953 raft_consensus.cc:729] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260716538087475d85549c9721ba881b, State: Initialized, Role: FOLLOWER
I20260619 20:49:47.708524 4953 consensus_queue.cc:260] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:47.708650 4957 tablet_bootstrap.cc:492] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Bootstrap starting.
I20260619 20:49:47.708827 4953 ts_tablet_manager.cc:1428] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:49:47.708841 4670 heartbeater.cc:503] Master 127.0.98.62:42185 was elected leader, sending a full tablet report...
I20260619 20:49:47.709358 4957 tablet_bootstrap.cc:654] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:47.709520 4954 tablet_bootstrap.cc:492] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: No bootstrap required, opened a new log
I20260619 20:49:47.709589 4954 ts_tablet_manager.cc:1397] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 20:49:47.709961 4957 log.cc:824] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:47.710738 4957 tablet_bootstrap.cc:492] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: No bootstrap required, opened a new log
I20260619 20:49:47.710803 4957 ts_tablet_manager.cc:1397] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:47.711694 4954 raft_consensus.cc:348] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:47.711845 4954 raft_consensus.cc:374] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:47.711879 4954 raft_consensus.cc:729] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6954f8f2fa84a02acb153e68f6d2904, State: Initialized, Role: FOLLOWER
I20260619 20:49:47.711997 4954 consensus_queue.cc:260] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:47.712301 4954 ts_tablet_manager.cc:1428] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:49:47.712306 4936 heartbeater.cc:503] Master 127.0.98.62:42185 was elected leader, sending a full tablet report...
I20260619 20:49:47.712823 4957 raft_consensus.cc:348] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:47.712934 4957 raft_consensus.cc:374] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:47.712980 4957 raft_consensus.cc:729] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38a58c7b98f4ebf8c5c3fde1b352028, State: Initialized, Role: FOLLOWER
I20260619 20:49:47.713088 4957 consensus_queue.cc:260] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:47.713393 4957 ts_tablet_manager.cc:1428] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:49:47.713393 4803 heartbeater.cc:503] Master 127.0.98.62:42185 was elected leader, sending a full tablet report...
W20260619 20:49:47.779026 4671 tablet.cc:2369] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:49:47.922610 4937 tablet.cc:2369] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:49:47.931578 4804 tablet.cc:2369] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:48.015703 4958 raft_consensus.cc:482] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:48.015867 4958 raft_consensus.cc:504] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:48.016219 4958 leader_election.cc:288] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a38a58c7b98f4ebf8c5c3fde1b352028 (127.0.98.2:39511), a6954f8f2fa84a02acb153e68f6d2904 (127.0.98.3:39229)
I20260619 20:49:48.020427 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.016698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51025 (local address 127.0.98.3:39229)
0619 20:49:48.016741 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.016743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.016752 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:48.016835 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.016836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.016848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.016901 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:48.017075 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.017866 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.019445 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.020025 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.020101 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.020111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.020319 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.020354 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:49:48.020427 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.016480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48597 (local address 127.0.98.2:39511)
0619 20:49:48.016548 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.016551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.016645 (+ 94us) server_negotiation.cc:407] Connection header received
0619 20:49:48.016806 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.016807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.016819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.016879 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:48.017068 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.017878 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.019451 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.020025 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.020101 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.020111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.020320 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.020354 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:49:48.020466 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.016608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39229 (local address 127.0.98.1:51025)
0619 20:49:48.016702 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.016712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.016763 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.016922 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.016922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.016923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:48.017001 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:48.017002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.017959 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.017959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:48.019357 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:48.019359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.020145 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.020147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:48.020230 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.020237 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.020307 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.020356 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":56,"thread_start_us":44,"threads_started":1}
I20260619 20:49:48.020470 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.016392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39511 (local address 127.0.98.1:48597)
0619 20:49:48.016558 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.016570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.016668 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.016913 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.016914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.016916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:48.016976 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:48.016979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.017958 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.017959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:48.019357 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:48.019359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.020146 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.020147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:48.020231 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.020235 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.020316 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.020356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":88,"thread_start_us":92,"threads_started":1}
I20260619 20:49:48.020838 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36000d82623243c8b1d7ed89e5f46f11" candidate_uuid: "260716538087475d85549c9721ba881b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6954f8f2fa84a02acb153e68f6d2904" is_pre_election: true
I20260619 20:49:48.020999 4890 raft_consensus.cc:2454] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 260716538087475d85549c9721ba881b in term 0.
I20260619 20:49:48.021117 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36000d82623243c8b1d7ed89e5f46f11" candidate_uuid: "260716538087475d85549c9721ba881b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" is_pre_election: true
I20260619 20:49:48.021219 4557 leader_election.cc:302] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 260716538087475d85549c9721ba881b, a6954f8f2fa84a02acb153e68f6d2904; no voters:
I20260619 20:49:48.021288 4757 raft_consensus.cc:2454] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 260716538087475d85549c9721ba881b in term 0.
I20260619 20:49:48.021373 4958 raft_consensus.cc:2781] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:48.021437 4958 raft_consensus.cc:482] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:48.021481 4958 raft_consensus.cc:3037] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.022137 4958 raft_consensus.cc:504] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:48.022270 4958 leader_election.cc:288] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [CANDIDATE]: Term 1 election: Requested vote from peers a38a58c7b98f4ebf8c5c3fde1b352028 (127.0.98.2:39511), a6954f8f2fa84a02acb153e68f6d2904 (127.0.98.3:39229)
I20260619 20:49:48.022439 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36000d82623243c8b1d7ed89e5f46f11" candidate_uuid: "260716538087475d85549c9721ba881b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028"
I20260619 20:49:48.022467 4890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36000d82623243c8b1d7ed89e5f46f11" candidate_uuid: "260716538087475d85549c9721ba881b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6954f8f2fa84a02acb153e68f6d2904"
I20260619 20:49:48.022531 4757 raft_consensus.cc:3037] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.022534 4890 raft_consensus.cc:3037] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.023170 4757 raft_consensus.cc:2454] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 260716538087475d85549c9721ba881b in term 1.
I20260619 20:49:48.023311 4890 raft_consensus.cc:2454] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 260716538087475d85549c9721ba881b in term 1.
I20260619 20:49:48.023389 4557 leader_election.cc:302] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 260716538087475d85549c9721ba881b, a38a58c7b98f4ebf8c5c3fde1b352028; no voters:
I20260619 20:49:48.023473 4958 raft_consensus.cc:2781] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:48.023615 4958 raft_consensus.cc:686] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [term 1 LEADER]: Becoming Leader. State: Replica: 260716538087475d85549c9721ba881b, State: Running, Role: LEADER
I20260619 20:49:48.023737 4958 consensus_queue.cc:237] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } }
I20260619 20:49:48.024550 4480 catalog_manager.cc:5495] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b reported cstate change: term changed from 0 to 1, leader changed from <none> to 260716538087475d85549c9721ba881b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "260716538087475d85549c9721ba881b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "260716538087475d85549c9721ba881b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } health_report { overall_health: UNKNOWN } } }
I20260619 20:49:48.053581 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.050090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.0.98.1:46351)
0619 20:49:48.050129 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.050132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.050147 (+ 15us) server_negotiation.cc:407] Connection header received
0619 20:49:48.050258 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.050259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.050280 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.050375 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:48.050547 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.051415 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.052583 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.052922 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.052990 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.053187 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:48.053344 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:48.053431 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.053473 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.053495 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:48.056408 4890 raft_consensus.cc:1261] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 1 FOLLOWER]: Refusing update from remote peer 260716538087475d85549c9721ba881b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:48.056746 4757 raft_consensus.cc:1261] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 1 FOLLOWER]: Refusing update from remote peer 260716538087475d85549c9721ba881b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:48.056932 4964 consensus_queue.cc:1035] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6954f8f2fa84a02acb153e68f6d2904" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:48.057047 4958 consensus_queue.cc:1035] T 36000d82623243c8b1d7ed89e5f46f11 P 260716538087475d85549c9721ba881b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a38a58c7b98f4ebf8c5c3fde1b352028" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:48.059243 4967 mvcc.cc:205] Tried to move back new op lower bound from 7298671362274803712 to 7298671362145898496. Current Snapshot: MvccSnapshot[applied={T|T < 7298671362274803712}]
I20260619 20:49:48.059597 4968 mvcc.cc:205] Tried to move back new op lower bound from 7298671362274803712 to 7298671362145898496. Current Snapshot: MvccSnapshot[applied={T|T < 7298671362274803712}]
I20260619 20:49:48.060145 4969 mvcc.cc:205] Tried to move back new op lower bound from 7298671362274803712 to 7298671362145898496. Current Snapshot: MvccSnapshot[applied={T|T < 7298671362274803712 or (T in {7298671362274803712})}]
I20260619 20:49:48.061426 4480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:48.061513 4480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:48.062180 4480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be881e29da704f60b03cf7f58e4b8318: Sending DeleteTablet for 3 replicas of tablet 36000d82623243c8b1d7ed89e5f46f11
I20260619 20:49:48.062588 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 36000d82623243c8b1d7ed89e5f46f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58988
I20260619 20:49:48.062589 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 36000d82623243c8b1d7ed89e5f46f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40988
I20260619 20:49:48.062908 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 36000d82623243c8b1d7ed89e5f46f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36542
I20260619 20:49:48.062958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4542
I20260619 20:49:48.065413 4977 tablet_replica.cc:330] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: stopping tablet replica
I20260619 20:49:48.065523 4977 raft_consensus.cc:2229] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:48.065658 4977 raft_consensus.cc:2258] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:48.065842 4976 tablet_replica.cc:330] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: stopping tablet replica
I20260619 20:49:48.065934 4976 raft_consensus.cc:2229] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:48.066061 4976 raft_consensus.cc:2258] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:48.066115 4977 ts_tablet_manager.cc:1905] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:48.066511 4976 ts_tablet_manager.cc:1905] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:48.067903 4977 ts_tablet_manager.cc:1918] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:48.067991 4977 log.cc:1196] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/36000d82623243c8b1d7ed89e5f46f11
I20260619 20:49:48.068140 4976 ts_tablet_manager.cc:1918] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:48.068210 4976 log.cc:1196] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/36000d82623243c8b1d7ed89e5f46f11
I20260619 20:49:48.068301 4977 ts_tablet_manager.cc:1939] T 36000d82623243c8b1d7ed89e5f46f11 P a38a58c7b98f4ebf8c5c3fde1b352028: Deleting consensus metadata
I20260619 20:49:48.068512 4976 ts_tablet_manager.cc:1939] T 36000d82623243c8b1d7ed89e5f46f11 P a6954f8f2fa84a02acb153e68f6d2904: Deleting consensus metadata
I20260619 20:49:48.068756 4464 catalog_manager.cc:4841] TS a38a58c7b98f4ebf8c5c3fde1b352028 (127.0.98.2:39511): tablet 36000d82623243c8b1d7ed89e5f46f11 (table test-table [id=76a32e5fe48c46adb61680d37037256d]) successfully deleted
I20260619 20:49:48.068856 4464 catalog_manager.cc:4841] TS a6954f8f2fa84a02acb153e68f6d2904 (127.0.98.3:39229): tablet 36000d82623243c8b1d7ed89e5f46f11 (table test-table [id=76a32e5fe48c46adb61680d37037256d]) successfully deleted
W20260619 20:49:48.068989 4467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46351 (error 108)
I20260619 20:49:48.069196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4675
I20260619 20:49:48.069963 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.069368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46351 (local address 127.0.0.1:36560)
0619 20:49:48.069886 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.069925 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495}
W20260619 20:49:48.073854 4467 catalog_manager.cc:4584] TS 260716538087475d85549c9721ba881b (127.0.98.1:46351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36000d82623243c8b1d7ed89e5f46f11: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46351: connect: Connection refused (error 111)
I20260619 20:49:48.073999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4808
I20260619 20:49:48.078401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4449
2026-06-19T20:49:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2253 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T20:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:48Z Disabled control of system clock
I20260619 20:49:48.104480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38241
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:49:48.182524 4982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:48.182711 4982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:48.182731 4982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:48.183761 4982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:48.183792 4982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:48.183804 4982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:48.183817 4982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:48.183830 4982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:48.185043 4982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:48.185217 4982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:48.185497 4982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:48.190011 4988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.190141 4982 server_base.cc:1034] running on GCE node
W20260619 20:49:48.190023 4990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.190042 4987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.190424 4982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:48.190673 4982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:48.191849 4982 hybrid_clock.cc:648] HybridClock initialized: now 1781902188191854 us; error 46 us; skew 500 ppm
I20260619 20:49:48.193046 4982 webserver.cc:460] Webserver started at http://127.0.98.62:37871/ using document root <none> and password file <none>
I20260619 20:49:48.193274 4982 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:48.193393 4982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:48.193555 4982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:48.194592 4982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "adf8b19a030a43c0af809e144d1fcd43"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "05b41eb98f4f404a2909c15395feb838"
server_key_iv: "0c348ba15183811eba0b7bc58f14b459"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.194936 4982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "adf8b19a030a43c0af809e144d1fcd43"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "05b41eb98f4f404a2909c15395feb838"
server_key_iv: "0c348ba15183811eba0b7bc58f14b459"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.196166 4982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:49:48.196867 4996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.197036 4982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:48.197130 4982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "adf8b19a030a43c0af809e144d1fcd43"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "05b41eb98f4f404a2909c15395feb838"
server_key_iv: "0c348ba15183811eba0b7bc58f14b459"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.197214 4982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:48.200297 4982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:48.200553 4982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:48.200708 4982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:48.204459 4982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38241
I20260619 20:49:48.204488 5048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38241 every 8 connection(s)
I20260619 20:49:48.204792 4982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:48.205426 5049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:48.208063 5049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43: Bootstrap starting.
I20260619 20:49:48.208348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 4982
I20260619 20:49:48.208452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:48.208616 392 external_mini_cluster.cc:1428] Setting key 2f9e3493a5656a600323eb79bfd49212
I20260619 20:49:48.208665 5049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:48.209017 5049 log.cc:824] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:48.209836 5049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43: No bootstrap required, opened a new log
I20260619 20:49:48.211074 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.209058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.0.98.62:38241)
0619 20:49:48.209253 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.209261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.209323 (+ 62us) server_negotiation.cc:407] Connection header received
0619 20:49:48.209816 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.209825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.209873 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.210021 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.210653 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.210657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.210657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.210661 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.210710 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.210722 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.210814 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.210845 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.210896 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260619 20:49:48.211666 5049 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf8b19a030a43c0af809e144d1fcd43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } }
I20260619 20:49:48.211854 5049 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:48.211898 5049 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf8b19a030a43c0af809e144d1fcd43, State: Initialized, Role: FOLLOWER
I20260619 20:49:48.212029 5049 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf8b19a030a43c0af809e144d1fcd43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } }
I20260619 20:49:48.212106 5049 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:48.212147 5049 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:48.212194 5049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.212986 5049 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf8b19a030a43c0af809e144d1fcd43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } }
I20260619 20:49:48.213102 5049 leader_election.cc:302] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf8b19a030a43c0af809e144d1fcd43; no voters:
I20260619 20:49:48.213245 5049 leader_election.cc:288] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:48.213336 5054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:48.213485 5054 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [term 1 LEADER]: Becoming Leader. State: Replica: adf8b19a030a43c0af809e144d1fcd43, State: Running, Role: LEADER
I20260619 20:49:48.213521 5049 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:48.213634 5054 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf8b19a030a43c0af809e144d1fcd43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } }
I20260619 20:49:48.213965 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf8b19a030a43c0af809e144d1fcd43. Latest consensus state: current_term: 1 leader_uuid: "adf8b19a030a43c0af809e144d1fcd43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf8b19a030a43c0af809e144d1fcd43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } }
I20260619 20:49:48.213961 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf8b19a030a43c0af809e144d1fcd43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf8b19a030a43c0af809e144d1fcd43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } }
I20260619 20:49:48.214139 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:48.214133 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:48.214357 5063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:48.214872 5063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:48.216210 5063 catalog_manager.cc:1347] Generated new cluster ID: 2279b0a3d4b64148972ccb603cf7d8b8
I20260619 20:49:48.216260 5063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:48.233788 5063 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:48.233963 5063 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:48.237618 5063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43: Generated new TSK 0
I20260619 20:49:48.237811 5063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:49:48.244506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38241
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:49:48.325398 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:48.325598 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:48.325620 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:48.326781 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:48.326817 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:48.326843 5073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:48.328152 5073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:48.328372 5073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:48.328704 5073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:48.334385 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.334353 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.334545 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.334456 5073 server_base.cc:1034] running on GCE node
I20260619 20:49:48.334769 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:48.335038 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:48.336189 5073 hybrid_clock.cc:648] HybridClock initialized: now 1781902188336178 us; error 26 us; skew 500 ppm
I20260619 20:49:48.337551 5073 webserver.cc:460] Webserver started at http://127.0.98.1:45115/ using document root <none> and password file <none>
I20260619 20:49:48.337803 5073 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:48.337849 5073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:48.337985 5073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:48.339118 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "d48a3b72c23e452a807f3ec34e2a6ebc"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "dfbb8c10447abd48ff5a559f0e24783a"
server_key_iv: "7fe6a10071e66d775880e3c28a7d348c"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.339479 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d48a3b72c23e452a807f3ec34e2a6ebc"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "dfbb8c10447abd48ff5a559f0e24783a"
server_key_iv: "7fe6a10071e66d775880e3c28a7d348c"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.340809 5073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:49:48.341614 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.341799 5073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:48.341882 5073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "d48a3b72c23e452a807f3ec34e2a6ebc"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "dfbb8c10447abd48ff5a559f0e24783a"
server_key_iv: "7fe6a10071e66d775880e3c28a7d348c"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.341934 5073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:48.345098 5073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:48.345356 5073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:48.345465 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:48.345631 5073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:48.345870 5073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:48.345898 5073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.345917 5073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:48.345929 5073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.351169 5073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34077
I20260619 20:49:48.351253 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34077 every 8 connection(s)
I20260619 20:49:48.351543 5073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:48.354476 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.352218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36873 (local address 127.0.98.62:38241)
0619 20:49:48.352281 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.352283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.352506 (+ 223us) server_negotiation.cc:407] Connection header received
0619 20:49:48.353031 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.353032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.353039 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.353090 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.354211 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.354213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.354213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.354218 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.354233 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.354237 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.354303 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.354393 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.354407 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:49:48.354662 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.352213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38241 (local address 127.0.98.1:36873)
0619 20:49:48.352390 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.352421 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.352916 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.353162 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.353166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.353173 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.354064 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.354065 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.354092 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.354117 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.354303 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.354308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.354311 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.354403 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.354460 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260619 20:49:48.355222 5201 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38241
I20260619 20:49:48.355320 5201 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:48.355463 5201 heartbeater.cc:511] Master 127.0.98.62:38241 requested a full tablet report, sending...
I20260619 20:49:48.355957 5013 ts_manager.cc:194] Registered new tserver with Master: d48a3b72c23e452a807f3ec34e2a6ebc (127.0.98.1:34077)
I20260619 20:49:48.359050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5073
I20260619 20:49:48.359136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:48.359320 392 external_mini_cluster.cc:1428] Setting key f591a63a6e509762d5707fb5240e5210
I20260619 20:49:48.360162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38241
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:49:48.438679 5204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:48.439050 5204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:48.439077 5204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:48.440274 5204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:48.440310 5204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:48.440335 5204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:48.441679 5204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:48.441859 5204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:48.442090 5204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:48.447459 5210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.447551 5204 server_base.cc:1034] running on GCE node
W20260619 20:49:48.447690 5212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.447853 5209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.448125 5204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:48.448423 5204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:48.449609 5204 hybrid_clock.cc:648] HybridClock initialized: now 1781902188449595 us; error 28 us; skew 500 ppm
I20260619 20:49:48.450814 5204 webserver.cc:460] Webserver started at http://127.0.98.2:35563/ using document root <none> and password file <none>
I20260619 20:49:48.451012 5204 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:48.451056 5204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:48.451159 5204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:48.452275 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e865861fddb4feca1a3d57bbf14aa58"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "2a0febc755743efc453b16b0d157c0e8"
server_key_iv: "59df9cdd3a409bfb32ce65d1fb5a1e32"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.452658 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e865861fddb4feca1a3d57bbf14aa58"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "2a0febc755743efc453b16b0d157c0e8"
server_key_iv: "59df9cdd3a409bfb32ce65d1fb5a1e32"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.454103 5204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:49:48.454892 5218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.455062 5204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:48.455165 5204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "0e865861fddb4feca1a3d57bbf14aa58"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "2a0febc755743efc453b16b0d157c0e8"
server_key_iv: "59df9cdd3a409bfb32ce65d1fb5a1e32"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.455242 5204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:48.458642 5204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:48.458902 5204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:48.459046 5204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:48.459245 5204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:48.459559 5204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:48.459619 5204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.459661 5204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:48.459693 5204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.465688 5204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45113
I20260619 20:49:48.465754 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45113 every 8 connection(s)
I20260619 20:49:48.466079 5204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:48.469067 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.466886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58079 (local address 127.0.98.62:38241)
0619 20:49:48.466937 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.466939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.467180 (+ 241us) server_negotiation.cc:407] Connection header received
0619 20:49:48.467657 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.467658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.467664 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.467720 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.468821 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.468823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.468823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.468828 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.468844 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.468848 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.468907 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.468983 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.468996 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:49:48.469246 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.466892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38241 (local address 127.0.98.2:58079)
0619 20:49:48.467071 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.467101 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.467556 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.467787 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.467791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.467801 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.468678 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.468678 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.468705 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.468730 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.468917 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.468922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.468924 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.468994 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.469051 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260619 20:49:48.469727 5332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38241
I20260619 20:49:48.469794 5332 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:48.469954 5332 heartbeater.cc:511] Master 127.0.98.62:38241 requested a full tablet report, sending...
I20260619 20:49:48.470284 5013 ts_manager.cc:194] Registered new tserver with Master: 0e865861fddb4feca1a3d57bbf14aa58 (127.0.98.2:45113)
I20260619 20:49:48.474480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5204
I20260619 20:49:48.474555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:48.474758 392 external_mini_cluster.cc:1428] Setting key 0025c1ed7f5e14d66f113c9afb7deac2
I20260619 20:49:48.475605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38241
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:49:48.553033 5335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:48.553220 5335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:48.553241 5335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:48.554296 5335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:48.554328 5335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:48.554353 5335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:48.555629 5335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:48.555801 5335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:48.556026 5335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:48.560985 5343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.561043 5335 server_base.cc:1034] running on GCE node
W20260619 20:49:48.561012 5341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.560986 5340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.561419 5335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:48.561616 5335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:48.562752 5335 hybrid_clock.cc:648] HybridClock initialized: now 1781902188562740 us; error 28 us; skew 500 ppm
I20260619 20:49:48.563864 5335 webserver.cc:460] Webserver started at http://127.0.98.3:46157/ using document root <none> and password file <none>
I20260619 20:49:48.564079 5335 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:48.564121 5335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:48.564204 5335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:48.565191 5335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9803045f91d40b8861285dab68b55cb"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "f8140e4f993d578db7d45dfd93b2d62c"
server_key_iv: "03fc28668591440eaa04174b15eb7977"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.565569 5335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9803045f91d40b8861285dab68b55cb"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "f8140e4f993d578db7d45dfd93b2d62c"
server_key_iv: "03fc28668591440eaa04174b15eb7977"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.567191 5335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:48.567899 5349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.568069 5335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:48.568140 5335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "e9803045f91d40b8861285dab68b55cb"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "f8140e4f993d578db7d45dfd93b2d62c"
server_key_iv: "03fc28668591440eaa04174b15eb7977"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.568190 5335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:48.571512 5335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:48.571713 5335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:48.571801 5335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:48.571955 5335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:48.572176 5335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:48.572203 5335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.572223 5335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:48.572235 5335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.577487 5335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36635
I20260619 20:49:48.577564 5462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36635 every 8 connection(s)
I20260619 20:49:48.577877 5335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:48.580422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5335
I20260619 20:49:48.580533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:48.580727 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.578607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34097 (local address 127.0.98.62:38241)
0619 20:49:48.578654 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.578656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.578874 (+ 218us) server_negotiation.cc:407] Connection header received
0619 20:49:48.579281 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.579281 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.579286 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.579344 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.580497 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.580499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.580500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.580505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.580522 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.580527 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.580585 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.580661 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.580675 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:48.580750 392 external_mini_cluster.cc:1428] Setting key d23e2465b3177da79dfe77d7b998fc06
I20260619 20:49:48.580973 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.578571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38241 (local address 127.0.98.3:34097)
0619 20:49:48.578741 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.578779 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.579205 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.579392 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.579395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.579402 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.580308 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.580309 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.580347 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.580383 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.580586 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.580591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.580593 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.580679 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.580742 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260619 20:49:48.581537 5463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38241
I20260619 20:49:48.581629 5463 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:48.581772 5463 heartbeater.cc:511] Master 127.0.98.62:38241 requested a full tablet report, sending...
I20260619 20:49:48.582145 5013 ts_manager.cc:194] Registered new tserver with Master: e9803045f91d40b8861285dab68b55cb (127.0.98.3:36635)
I20260619 20:49:48.592578 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:49:48.594807 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.593879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.0.98.62:38241)
0619 20:49:48.593943 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.593946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.594035 (+ 89us) server_negotiation.cc:407] Connection header received
0619 20:49:48.594085 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.594085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.594089 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.594132 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.594612 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.594613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.594613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.594616 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.594628 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.594631 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.594668 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.594703 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.594713 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:49:48.596114 5013 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:48.597713 5013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:48.601588 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.599810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45113 (local address 127.0.0.1:48538)
0619 20:49:48.599973 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.600010 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.600064 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.600807 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.600808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.600810 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.601141 (+ 331us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.601141 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.601146 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.601156 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.601423 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.601429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.601432 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.601475 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.601489 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260619 20:49:48.601612 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.599971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.0.98.3:36635)
0619 20:49:48.600227 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.600231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.600241 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:48.600262 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.600265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.600274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.600343 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.601325 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.601328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.601328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.601332 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.601357 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.601372 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.601434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.601511 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.601526 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260619 20:49:48.601707 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.599996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.0.98.2:45113)
0619 20:49:48.600642 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.600646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.600655 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:48.600679 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.600681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.600689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.600802 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.601234 (+ 432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.601240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.601241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.601245 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.601276 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.601326 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.601581 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.601601 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.601617 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":80,"threads_started":1}
I20260619 20:49:48.601588 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.599813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36635 (local address 127.0.0.1:42026)
0619 20:49:48.600121 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.600131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.600173 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.600360 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.600364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.600370 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.600833 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.600834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.600848 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.600872 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.601438 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.601439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.601440 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.601496 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.601509 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":93,"threads_started":1}
I20260619 20:49:48.601935 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.600050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.0.98.1:34077)
0619 20:49:48.600199 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.600203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.600233 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:49:48.600303 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.600304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.600311 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.600412 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.601337 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.601341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.601342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.601346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.601381 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.601394 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.601460 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.601812 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.601829 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260619 20:49:48.602046 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.599836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.0.1:52238)
0619 20:49:48.600168 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.600180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.600242 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.600423 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.600424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.600427 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.601013 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.601014 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.601020 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.601031 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.601486 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.601486 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.601487 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.601531 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.601541 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":88,"threads_started":1}
I20260619 20:49:48.602347 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 (DEFAULT_TABLE table=test-table [id=4ab6031cb0f24f628a3d5c7e537c6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:48.602720 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:48.602699 5397 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 (DEFAULT_TABLE table=test-table [id=4ab6031cb0f24f628a3d5c7e537c6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:48.602998 5397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:48.605738 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 (DEFAULT_TABLE table=test-table [id=4ab6031cb0f24f628a3d5c7e537c6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:48.606010 5477 tablet_bootstrap.cc:492] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Bootstrap starting.
I20260619 20:49:48.606102 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:48.606611 5477 tablet_bootstrap.cc:654] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:48.606897 5477 log.cc:824] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:48.607685 5477 tablet_bootstrap.cc:492] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: No bootstrap required, opened a new log
I20260619 20:49:48.607753 5477 ts_tablet_manager.cc:1397] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:49:48.608795 5479 tablet_bootstrap.cc:492] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: Bootstrap starting.
I20260619 20:49:48.609269 5480 tablet_bootstrap.cc:492] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Bootstrap starting.
I20260619 20:49:48.609457 5479 tablet_bootstrap.cc:654] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:48.609822 5480 tablet_bootstrap.cc:654] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:48.609803 5479 log.cc:824] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:48.610155 5480 log.cc:824] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:48.610117 5477 raft_consensus.cc:348] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.610248 5477 raft_consensus.cc:374] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:48.610292 5477 raft_consensus.cc:729] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e865861fddb4feca1a3d57bbf14aa58, State: Initialized, Role: FOLLOWER
I20260619 20:49:48.610426 5477 consensus_queue.cc:260] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.610630 5479 tablet_bootstrap.cc:492] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: No bootstrap required, opened a new log
I20260619 20:49:48.610689 5477 ts_tablet_manager.cc:1428] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:49:48.610690 5479 ts_tablet_manager.cc:1397] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:48.610822 5332 heartbeater.cc:503] Master 127.0.98.62:38241 was elected leader, sending a full tablet report...
I20260619 20:49:48.610863 5480 tablet_bootstrap.cc:492] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: No bootstrap required, opened a new log
I20260619 20:49:48.610913 5480 ts_tablet_manager.cc:1397] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:48.612911 5479 raft_consensus.cc:348] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.613087 5479 raft_consensus.cc:374] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:48.613134 5479 raft_consensus.cc:729] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48a3b72c23e452a807f3ec34e2a6ebc, State: Initialized, Role: FOLLOWER
I20260619 20:49:48.613085 5480 raft_consensus.cc:348] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.613240 5479 consensus_queue.cc:260] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.613277 5480 raft_consensus.cc:374] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:48.613404 5480 raft_consensus.cc:729] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9803045f91d40b8861285dab68b55cb, State: Initialized, Role: FOLLOWER
I20260619 20:49:48.613509 5479 ts_tablet_manager.cc:1428] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 20:49:48.613510 5480 consensus_queue.cc:260] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.613684 5201 heartbeater.cc:503] Master 127.0.98.62:38241 was elected leader, sending a full tablet report...
I20260619 20:49:48.613786 5480 ts_tablet_manager.cc:1428] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:49:48.613807 5463 heartbeater.cc:503] Master 127.0.98.62:38241 was elected leader, sending a full tablet report...
W20260619 20:49:48.716595 5333 tablet.cc:2369] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:48.719888 5483 raft_consensus.cc:482] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:48.720006 5483 raft_consensus.cc:504] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.720358 5483 leader_election.cc:288] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9803045f91d40b8861285dab68b55cb (127.0.98.3:36635), d48a3b72c23e452a807f3ec34e2a6ebc (127.0.98.1:34077)
I20260619 20:49:48.721767 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.720539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.98.2:33037)
0619 20:49:48.720594 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.720606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.720642 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.720858 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.720859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.720862 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.721411 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.721411 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.721418 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.721429 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.721621 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.721622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.721623 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.721675 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.721689 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:49:48.721908 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.720612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33037 (local address 127.0.98.1:34077)
0619 20:49:48.720662 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.720672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.720682 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:48.720713 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.720714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.720729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.720810 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.721512 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.721514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.721514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.721519 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.721535 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.721541 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.721622 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.721815 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.721829 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:49:48.721954 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.720556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36635 (local address 127.0.98.2:47219)
0619 20:49:48.720731 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.720742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.720796 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.720955 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.720955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.720957 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:48.721599 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:49:48.721600 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:48.721607 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:49:48.721618 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:48.721784 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:48.721784 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:48.721785 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:49:48.721858 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:49:48.721873 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260619 20:49:48.721941 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.720629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47219 (local address 127.0.98.3:36635)
0619 20:49:48.720696 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.720698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.720833 (+ 135us) server_negotiation.cc:407] Connection header received
0619 20:49:48.720862 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.720863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.720870 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.720951 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.721680 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.721681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.721681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.721686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.721700 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.721704 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.721786 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.721861 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.721877 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:49:48.722194 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3a8527000c4ceeaa6d71ec272ed7d1" candidate_uuid: "0e865861fddb4feca1a3d57bbf14aa58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" is_pre_election: true
I20260619 20:49:48.722376 5155 raft_consensus.cc:2454] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e865861fddb4feca1a3d57bbf14aa58 in term 0.
I20260619 20:49:48.722469 5417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3a8527000c4ceeaa6d71ec272ed7d1" candidate_uuid: "0e865861fddb4feca1a3d57bbf14aa58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9803045f91d40b8861285dab68b55cb" is_pre_election: true
I20260619 20:49:48.722636 5417 raft_consensus.cc:2454] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e865861fddb4feca1a3d57bbf14aa58 in term 0.
I20260619 20:49:48.722625 5220 leader_election.cc:302] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e865861fddb4feca1a3d57bbf14aa58, d48a3b72c23e452a807f3ec34e2a6ebc; no voters:
I20260619 20:49:48.722781 5483 raft_consensus.cc:2781] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:48.722880 5483 raft_consensus.cc:482] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:48.722910 5483 raft_consensus.cc:3037] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.723634 5483 raft_consensus.cc:504] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.723780 5483 leader_election.cc:288] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [CANDIDATE]: Term 1 election: Requested vote from peers e9803045f91d40b8861285dab68b55cb (127.0.98.3:36635), d48a3b72c23e452a807f3ec34e2a6ebc (127.0.98.1:34077)
I20260619 20:49:48.723912 5417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3a8527000c4ceeaa6d71ec272ed7d1" candidate_uuid: "0e865861fddb4feca1a3d57bbf14aa58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9803045f91d40b8861285dab68b55cb"
I20260619 20:49:48.723938 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3a8527000c4ceeaa6d71ec272ed7d1" candidate_uuid: "0e865861fddb4feca1a3d57bbf14aa58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc"
I20260619 20:49:48.723992 5417 raft_consensus.cc:3037] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.724047 5155 raft_consensus.cc:3037] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.724735 5155 raft_consensus.cc:2454] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e865861fddb4feca1a3d57bbf14aa58 in term 1.
I20260619 20:49:48.724784 5417 raft_consensus.cc:2454] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e865861fddb4feca1a3d57bbf14aa58 in term 1.
I20260619 20:49:48.724943 5220 leader_election.cc:302] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e865861fddb4feca1a3d57bbf14aa58, d48a3b72c23e452a807f3ec34e2a6ebc; no voters:
I20260619 20:49:48.725045 5483 raft_consensus.cc:2781] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:48.725214 5483 raft_consensus.cc:686] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 1 LEADER]: Becoming Leader. State: Replica: 0e865861fddb4feca1a3d57bbf14aa58, State: Running, Role: LEADER
I20260619 20:49:48.725342 5483 consensus_queue.cc:237] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } }
I20260619 20:49:48.726154 5013 catalog_manager.cc:5495] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e865861fddb4feca1a3d57bbf14aa58 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0e865861fddb4feca1a3d57bbf14aa58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e865861fddb4feca1a3d57bbf14aa58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 } health_report { overall_health: UNKNOWN } } }
I20260619 20:49:48.746295 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.745185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.0.98.2:45113)
0619 20:49:48.745254 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.745256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.745266 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:49:48.745418 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.745419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.745434 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.745506 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.746071 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.746072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.746072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.746076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.746091 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.746095 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.746147 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.746210 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.746223 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:49:48.748147 5155 raft_consensus.cc:1261] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 1 FOLLOWER]: Refusing update from remote peer 0e865861fddb4feca1a3d57bbf14aa58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:48.748149 5417 raft_consensus.cc:1261] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 1 FOLLOWER]: Refusing update from remote peer 0e865861fddb4feca1a3d57bbf14aa58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:48.748469 5487 consensus_queue.cc:1035] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9803045f91d40b8861285dab68b55cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:48.748605 5483 consensus_queue.cc:1035] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d48a3b72c23e452a807f3ec34e2a6ebc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:48.750814 5490 mvcc.cc:205] Tried to move back new op lower bound from 7298671365109878784 to 7298671365019729920. Current Snapshot: MvccSnapshot[applied={T|T < 7298671365109878784}]
I20260619 20:49:48.753163 5013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:48.753252 5013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:48.753983 5013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adf8b19a030a43c0af809e144d1fcd43: Sending DeleteTablet for 3 replicas of tablet 9c3a8527000c4ceeaa6d71ec272ed7d1
I20260619 20:49:48.754316 5266 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:48 UTC) from {username='slave'} at 127.0.0.1:48538
I20260619 20:49:48.754325 5397 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:48 UTC) from {username='slave'} at 127.0.0.1:42026
I20260619 20:49:48.754351 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:48 UTC) from {username='slave'} at 127.0.0.1:52238
I20260619 20:49:48.754587 5501 tablet_replica.cc:330] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc: stopping tablet replica
I20260619 20:49:48.754678 5501 raft_consensus.cc:2229] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:48.754776 5501 raft_consensus.cc:2258] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P d48a3b72c23e452a807f3ec34e2a6ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:48.754828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5073
I20260619 20:49:48.755498 5500 tablet_replica.cc:330] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: stopping tablet replica
I20260619 20:49:48.755584 5500 raft_consensus.cc:2229] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:48.755666 5500 raft_consensus.cc:2258] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:48.755872 5499 tablet_replica.cc:330] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: stopping tablet replica
I20260619 20:49:48.755960 5499 raft_consensus.cc:2229] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:48.756053 5499 raft_consensus.cc:2258] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:48.756276 5500 ts_tablet_manager.cc:1905] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:48.757201 5499 ts_tablet_manager.cc:1905] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:48.757865 5500 ts_tablet_manager.cc:1918] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:48.758003 5500 log.cc:1196] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/9c3a8527000c4ceeaa6d71ec272ed7d1
I20260619 20:49:48.758302 5500 ts_tablet_manager.cc:1939] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P 0e865861fddb4feca1a3d57bbf14aa58: Deleting consensus metadata
I20260619 20:49:48.758672 5000 catalog_manager.cc:4841] TS 0e865861fddb4feca1a3d57bbf14aa58 (127.0.98.2:45113): tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 (table test-table [id=4ab6031cb0f24f628a3d5c7e537c6bdc]) successfully deleted
I20260619 20:49:48.759179 5499 ts_tablet_manager.cc:1918] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:48.759272 5499 log.cc:1196] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/9c3a8527000c4ceeaa6d71ec272ed7d1
I20260619 20:49:48.759497 5499 ts_tablet_manager.cc:1939] T 9c3a8527000c4ceeaa6d71ec272ed7d1 P e9803045f91d40b8861285dab68b55cb: Deleting consensus metadata
I20260619 20:49:48.759860 4999 catalog_manager.cc:4841] TS e9803045f91d40b8861285dab68b55cb (127.0.98.3:36635): tablet 9c3a8527000c4ceeaa6d71ec272ed7d1 (table test-table [id=4ab6031cb0f24f628a3d5c7e537c6bdc]) successfully deleted
W20260619 20:49:48.762271 4998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34077 (error 108)
I20260619 20:49:48.762513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5204
I20260619 20:49:48.763090 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.762603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34077 (local address 127.0.0.1:52252)
0619 20:49:48.763011 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.763050 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390}
W20260619 20:49:48.763360 4998 catalog_manager.cc:4584] TS d48a3b72c23e452a807f3ec34e2a6ebc (127.0.98.1:34077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c3a8527000c4ceeaa6d71ec272ed7d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34077: connect: Connection refused (error 111)
I20260619 20:49:48.767711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5335
I20260619 20:49:48.772226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 4982
2026-06-19T20:49:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (693 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 20:49:48.791405 392 mini_oidc.cc:152] Starting JWKS server
I20260619 20:49:48.792390 392 webserver.cc:460] Webserver started at https://127.0.0.1:46323/ using document root <none> and password file <none>
I20260619 20:49:48.792461 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:49:48.792580 392 webserver.cc:460] Webserver started at http://127.0.0.1:42645/ using document root <none> and password file <none>
2026-06-19T20:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:48Z Disabled control of system clock
I20260619 20:49:48.800127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46327
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46323/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/testchainca.cert with env {}
W20260619 20:49:48.878713 5508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:48.878916 5508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:48.878937 5508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:48.880020 5508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:48.880054 5508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:48.880067 5508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:48.880079 5508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:48.880091 5508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:49:48.880103 5508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:49:48.880115 5508 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46323/jwks.json
I20260619 20:49:48.881567 5508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46323/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:48.881747 5508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:48.881907 5508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:49:48.886713 5508 server_base.cc:1034] running on GCE node
W20260619 20:49:48.886708 5513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.886689 5514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:48.886695 5516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:48.887184 5508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:48.887514 5508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:48.888684 5508 hybrid_clock.cc:648] HybridClock initialized: now 1781902188888673 us; error 39 us; skew 500 ppm
I20260619 20:49:48.889906 5508 webserver.cc:460] Webserver started at http://127.0.98.62:42767/ using document root <none> and password file <none>
I20260619 20:49:48.890152 5508 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:48.890231 5508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:48.890384 5508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:48.891460 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "0972220884794affaf36a69c7d8d2823"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "cb68a32d60e8e2fb41be531ae55016f9"
server_key_iv: "6c7b050f322e20064917081b8d3e2adc"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.891825 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "0972220884794affaf36a69c7d8d2823"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "cb68a32d60e8e2fb41be531ae55016f9"
server_key_iv: "6c7b050f322e20064917081b8d3e2adc"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.893362 5508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:49:48.894117 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:48.894304 5508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:48.894402 5508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "0972220884794affaf36a69c7d8d2823"
format_stamp: "Formatted at 2026-06-19 20:49:48 on dist-test-slave-0smx"
server_key: "cb68a32d60e8e2fb41be531ae55016f9"
server_key_iv: "6c7b050f322e20064917081b8d3e2adc"
server_key_version: "encryptionkey@0"
I20260619 20:49:48.894485 5508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:48.910113 5508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:48.910436 5508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:48.910596 5508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:48.914409 5508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46327
I20260619 20:49:48.914445 5574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46327 every 8 connection(s)
I20260619 20:49:48.914804 5508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:48.915849 5575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:48.918180 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823: Bootstrap starting.
I20260619 20:49:48.918727 5575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:48.918978 5575 log.cc:824] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:48.919651 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823: No bootstrap required, opened a new log
I20260619 20:49:48.921142 5575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0972220884794affaf36a69c7d8d2823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } }
I20260619 20:49:48.921281 5575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:48.921386 5575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0972220884794affaf36a69c7d8d2823, State: Initialized, Role: FOLLOWER
I20260619 20:49:48.921535 5575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0972220884794affaf36a69c7d8d2823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } }
I20260619 20:49:48.921615 5575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:48.921638 5575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:48.921666 5575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:48.922125 5575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0972220884794affaf36a69c7d8d2823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } }
I20260619 20:49:48.922204 5575 leader_election.cc:302] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0972220884794affaf36a69c7d8d2823; no voters:
I20260619 20:49:48.922329 5575 leader_election.cc:288] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:48.922410 5578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:48.922593 5578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [term 1 LEADER]: Becoming Leader. State: Replica: 0972220884794affaf36a69c7d8d2823, State: Running, Role: LEADER
I20260619 20:49:48.922621 5575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:48.922688 5578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0972220884794affaf36a69c7d8d2823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } }
I20260619 20:49:48.923056 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0972220884794affaf36a69c7d8d2823. Latest consensus state: current_term: 1 leader_uuid: "0972220884794affaf36a69c7d8d2823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0972220884794affaf36a69c7d8d2823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } }
I20260619 20:49:48.923151 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:48.923141 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0972220884794affaf36a69c7d8d2823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0972220884794affaf36a69c7d8d2823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } }
I20260619 20:49:48.923303 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:48.923812 5588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:48.924170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5508
I20260619 20:49:48.924279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:48.924306 5588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:48.924506 392 external_mini_cluster.cc:1428] Setting key e14289074ac2c8d16b947930cf7a3cd3
I20260619 20:49:48.928467 5588 catalog_manager.cc:1347] Generated new cluster ID: 78a8ab41246c42a6b3e349b7fd4abbb7
I20260619 20:49:48.928522 5588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:48.930218 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.925357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.0.98.62:46327)
0619 20:49:48.925603 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.925610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.925634 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:49:48.925724 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.925729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.925786 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.925902 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:48.926127 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.926974 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.928098 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.928666 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.928830 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.929512 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:48.929516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:48.929516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:48.929696 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:48.929727 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:48.929737 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:48.929810 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:48.929894 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 20:49:48.930058 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":128,"threads_started":1}
I20260619 20:49:48.930998 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:49:48.952783 5588 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:48.953384 5588 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:48.958278 5588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0972220884794affaf36a69c7d8d2823: Generated new TSK 0
I20260619 20:49:48.958492 5588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 20:49:48.991453 5605 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 20:49:48.991570 5605 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 20:49:48.991812 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:48.971674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.98.62:46327)
0619 20:49:48.971762 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:48.971765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:48.971904 (+ 139us) server_negotiation.cc:407] Connection header received
0619 20:49:48.972001 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:48.972002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:48.972014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:48.972079 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:49:48.972282 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.972767 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.973940 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.974129 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.974192 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.974406 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:49:48.991586 (+ 17180us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46323/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 20:49:48.991732 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46323/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 20:49:48.991882 5598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46323/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 20:49:48.992200 5604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:48.971610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46327 (local address 127.0.0.1:47106)
0619 20:49:48.971817 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:48.971829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:48.971908 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:48.972101 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:48.972102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:48.972104 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:49:48.972170 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:48.972172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.972869 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.972870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:48.973840 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:48.973843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:48.974204 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:48.974208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:48.974272 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:48.974291 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:49:48.991798 (+ 17507us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46323/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 20:49:48.991947 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46323/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":129,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:46327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46323/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 20:49:48.992654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5508
2026-06-19T20:49:49Z chronyd exiting
I20260619 20:49:49.028342 392 test_util.cc:175] -----------------------------------------------
I20260619 20:49:49.028421 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781902166366061-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 20:49:49.029094 392 mini_oidc.cc:152] Starting JWKS server
I20260619 20:49:49.029799 392 webserver.cc:460] Webserver started at https://127.0.0.1:44129/ using document root <none> and password file <none>
I20260619 20:49:49.029841 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:49:49.029961 392 webserver.cc:460] Webserver started at http://127.0.0.1:42243/ using document root <none> and password file <none>
2026-06-19T20:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:49Z Disabled control of system clock
I20260619 20:49:49.037274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37697
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37697
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44129/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/testchainca.cert with env {}
W20260619 20:49:49.115442 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:49.115633 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:49.115654 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:49.116717 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:49.116750 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:49.116763 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:49.116775 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:49.116787 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:49:49.116799 5611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:49:49.116811 5611 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44129/jwks.json
I20260619 20:49:49.118216 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44129/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:49.118403 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:49.118554 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:49.123150 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:49.123188 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:49.123183 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:49.123642 5611 server_base.cc:1034] running on GCE node
I20260619 20:49:49.123868 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:49.124107 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:49.125254 5611 hybrid_clock.cc:648] HybridClock initialized: now 1781902189125247 us; error 42 us; skew 500 ppm
I20260619 20:49:49.126398 5611 webserver.cc:460] Webserver started at http://127.0.98.62:36899/ using document root <none> and password file <none>
I20260619 20:49:49.126633 5611 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:49.126679 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:49.126822 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:49.127826 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "151ab226636249dfb69db86ee04a2105"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "162362b818f2fe47c5305ab06a143908"
server_key_iv: "3506fd894d31cc81ba6f70f5feb56498"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.128180 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "151ab226636249dfb69db86ee04a2105"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "162362b818f2fe47c5305ab06a143908"
server_key_iv: "3506fd894d31cc81ba6f70f5feb56498"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.129478 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:49.130155 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:49.130333 5611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:49.130419 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "151ab226636249dfb69db86ee04a2105"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "162362b818f2fe47c5305ab06a143908"
server_key_iv: "3506fd894d31cc81ba6f70f5feb56498"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.130491 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:49.139469 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:49.139721 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:49.139858 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:49.143824 5611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37697
I20260619 20:49:49.143888 5677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37697 every 8 connection(s)
I20260619 20:49:49.144158 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:49.145036 5678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:49.147356 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105: Bootstrap starting.
I20260619 20:49:49.147939 5678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:49.148255 5678 log.cc:824] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:49.148940 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105: No bootstrap required, opened a new log
I20260619 20:49:49.150604 5678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151ab226636249dfb69db86ee04a2105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37697 } }
I20260619 20:49:49.150751 5678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:49.150801 5678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151ab226636249dfb69db86ee04a2105, State: Initialized, Role: FOLLOWER
I20260619 20:49:49.150969 5678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151ab226636249dfb69db86ee04a2105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37697 } }
I20260619 20:49:49.151073 5678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:49.151119 5678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:49.151175 5678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:49.151208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5611
I20260619 20:49:49.151324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:49.151528 392 external_mini_cluster.cc:1428] Setting key 3c09489232d8d46def1a709a403e1322
I20260619 20:49:49.151948 5678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151ab226636249dfb69db86ee04a2105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37697 } }
I20260619 20:49:49.152062 5678 leader_election.cc:302] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 151ab226636249dfb69db86ee04a2105; no voters:
I20260619 20:49:49.152236 5678 leader_election.cc:288] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:49.152273 5683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:49.152446 5683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [term 1 LEADER]: Becoming Leader. State: Replica: 151ab226636249dfb69db86ee04a2105, State: Running, Role: LEADER
I20260619 20:49:49.152530 5678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:49.152581 5683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151ab226636249dfb69db86ee04a2105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37697 } }
I20260619 20:49:49.153874 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "151ab226636249dfb69db86ee04a2105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151ab226636249dfb69db86ee04a2105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37697 } } }
I20260619 20:49:49.153982 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:49.154165 5698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:49.154248 5685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 151ab226636249dfb69db86ee04a2105. Latest consensus state: current_term: 1 leader_uuid: "151ab226636249dfb69db86ee04a2105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151ab226636249dfb69db86ee04a2105" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37697 } } }
I20260619 20:49:49.154517 5685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:49.154762 5698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:49.156514 5698 catalog_manager.cc:1347] Generated new cluster ID: 468ac0629fa0476ca48e7bf8917dd5fd
I20260619 20:49:49.156589 5698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:49.158372 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:49.151955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.98.62:37697)
0619 20:49:49.152154 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:49.152161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:49.152189 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:49:49.152326 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:49.152333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:49.152429 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:49.152567 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:49.152847 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.154658 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.156011 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.156580 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.156694 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.157584 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:49.157587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:49.157588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:49.157786 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:49.157823 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:49.157836 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:49.157885 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:49.158112 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 20:49:49.158182 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260619 20:49:49.159258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:49:49.184782 5698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:49.185408 5698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:49.189038 5698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 151ab226636249dfb69db86ee04a2105: Generated new TSK 0
I20260619 20:49:49.189224 5698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 20:49:49.209779 5708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 20:49:49.209856 5708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 20:49:49.210165 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:49.196038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.98.62:37697)
0619 20:49:49.196082 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:49.196084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:49.196185 (+ 101us) server_negotiation.cc:407] Connection header received
0619 20:49:49.196238 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:49.196239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:49.196250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:49.196310 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:49:49.196455 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.196941 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.198086 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.198273 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.198320 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.198501 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:49:49.209997 (+ 11496us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 20:49:49.210116 (+ 119us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 20:49:49.210232 5682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 20:49:49.210340 5707 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:49.195937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37697 (local address 127.0.0.1:51892)
0619 20:49:49.196105 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:49.196116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:49.196179 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:49.196328 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:49.196329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:49.196330 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:49:49.196387 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:49.196389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.197040 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.197041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:49.197993 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:49.197996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.198346 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.198350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:49.198409 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.198414 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:49:49.210106 (+ 11692us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 20:49:49.210221 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37697: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260619 20:49:49.210628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5611
2026-06-19T20:49:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 20:49:49.247884 392 mini_oidc.cc:152] Starting JWKS server
I20260619 20:49:49.248574 392 webserver.cc:460] Webserver started at https://127.0.0.1:37231/ using document root <none> and password file <none>
I20260619 20:49:49.248615 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:49:49.248762 392 webserver.cc:460] Webserver started at http://127.0.0.1:33397/ using document root <none> and password file <none>
2026-06-19T20:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:49Z Disabled control of system clock
I20260619 20:49:49.256271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41679
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41679
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37231/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 20:49:49.334250 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:49.334492 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:49.334528 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:49.335672 5714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:49.335736 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:49.335757 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:49.335777 5714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:49.335807 5714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:49:49.335839 5714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:49:49.335868 5714 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37231/jwks.json
I20260619 20:49:49.337211 5714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37231/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:49.337512 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:49.337801 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:49:49.342592 5714 server_base.cc:1034] running on GCE node
W20260619 20:49:49.342680 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:49.342571 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:49.342561 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:49.343106 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:49.343428 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:49.344640 5714 hybrid_clock.cc:648] HybridClock initialized: now 1781902189344622 us; error 32 us; skew 500 ppm
I20260619 20:49:49.346220 5714 webserver.cc:460] Webserver started at http://127.0.98.62:46203/ using document root <none> and password file <none>
I20260619 20:49:49.346508 5714 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:49.346565 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:49.346751 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:49.347895 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "f5cb7d46df9a0d2751aac6398b13d646"
server_key_iv: "f1af89a3cecebe3104da58968755b255"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.348284 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "f5cb7d46df9a0d2751aac6398b13d646"
server_key_iv: "f1af89a3cecebe3104da58968755b255"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.349745 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:49.350504 5728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:49.350682 5714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:49.350787 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "f5cb7d46df9a0d2751aac6398b13d646"
server_key_iv: "f1af89a3cecebe3104da58968755b255"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.350872 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:49.379900 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:49.380266 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:49.380437 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:49.384654 5714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41679
I20260619 20:49:49.384724 5780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41679 every 8 connection(s)
I20260619 20:49:49.385103 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:49.385820 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:49.388398 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5: Bootstrap starting.
I20260619 20:49:49.388978 5781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:49.389283 5781 log.cc:824] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:49.390054 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5: No bootstrap required, opened a new log
I20260619 20:49:49.390329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5714
I20260619 20:49:49.390443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:49.390609 392 external_mini_cluster.cc:1428] Setting key dfe1576cf5b0270d7b80ec13a139fc6c
I20260619 20:49:49.392409 5781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41679 } }
I20260619 20:49:49.392587 5781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:49.392627 5781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8cbfd67f874f289e0cfc6e6a13acd5, State: Initialized, Role: FOLLOWER
I20260619 20:49:49.392776 5781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41679 } }
I20260619 20:49:49.392855 5781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:49.392890 5781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:49.392937 5781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:49.393637 5781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41679 } }
I20260619 20:49:49.393739 5781 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8cbfd67f874f289e0cfc6e6a13acd5; no voters:
I20260619 20:49:49.393869 5781 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:49.393950 5786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:49.394089 5781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:49.394313 5786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8cbfd67f874f289e0cfc6e6a13acd5, State: Running, Role: LEADER
I20260619 20:49:49.394419 5786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41679 } }
I20260619 20:49:49.394840 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8cbfd67f874f289e0cfc6e6a13acd5. Latest consensus state: current_term: 1 leader_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41679 } } }
I20260619 20:49:49.394851 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8cbfd67f874f289e0cfc6e6a13acd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41679 } } }
I20260619 20:49:49.394996 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:49.395004 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:49.395337 5797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:49.396207 5797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:49.397696 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:49.391132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.62:41679)
0619 20:49:49.391346 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:49.391355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:49.391389 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:49:49.391966 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:49.391975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:49.392054 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:49.392206 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:49.392368 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.394170 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.395409 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.395996 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.396129 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.396820 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:49.396831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:49.396831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:49.397072 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:49.397115 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:49.397129 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:49.397196 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:49.397356 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 20:49:49.397436 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260619 20:49:49.398254 5797 catalog_manager.cc:1347] Generated new cluster ID: 0f6b3351aaf34aafbff0f5c73aacd94d
I20260619 20:49:49.398308 5797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:49.398557 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:49:49.419163 5797 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:49.419871 5797 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:49.430384 5797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b8cbfd67f874f289e0cfc6e6a13acd5: Generated new TSK 0
I20260619 20:49:49.430626 5797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:49:49.440001 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:49.436241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.62:41679)
0619 20:49:49.436304 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:49.436307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:49.436366 (+ 59us) server_negotiation.cc:407] Connection header received
0619 20:49:49.436433 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:49.436435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:49.436449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:49.436545 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:49:49.436672 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.437472 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.438234 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.438542 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.438624 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.438759 (+ 135us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:49:49.439780 (+ 1021us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 20:49:49.439925 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 20:49:49.440066 5785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 20:49:49.441968 5810 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:49.436129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41679 (local address 127.0.0.1:37572)
0619 20:49:49.436312 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:49.436323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:49.436377 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:49.436546 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:49.436548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:49.436550 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:49:49.436604 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:49.436606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.437568 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.437569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:49.438155 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:49.438157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.438626 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.438629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:49.438668 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.438672 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:49:49.439889 (+ 1217us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 20:49:49.441868 (+ 1979us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1786,"thread_start_us":93,"threads_started":1}
I20260619 20:49:49.442317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5714
E20260619 20:49:49.453233 5811 webserver.cc:533] Webserver: sslize failed:
E20260619 20:49:49.453342 5811 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T20:49:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T20:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:49Z Disabled control of system clock
I20260619 20:49:49.488071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42813
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/testchainca.cert with env {}
W20260619 20:49:49.565637 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:49.565841 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:49.565864 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:49.566989 5815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:49.567026 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:49.567039 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:49.567051 5815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:49.567063 5815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:49:49.567075 5815 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:49:49.567086 5815 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 20:49:49.568485 5815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:49.568670 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:49.568825 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:49.573590 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:49.573611 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:49.573609 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:49.573879 5815 server_base.cc:1034] running on GCE node
I20260619 20:49:49.574131 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:49.574430 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:49.575604 5815 hybrid_clock.cc:648] HybridClock initialized: now 1781902189575589 us; error 25 us; skew 500 ppm
I20260619 20:49:49.576874 5815 webserver.cc:460] Webserver started at http://127.0.98.62:39305/ using document root <none> and password file <none>
I20260619 20:49:49.577162 5815 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:49.577229 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:49.577378 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:49.578403 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "068c5f4a63314a63bf3cd99324b4cfcb"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "c457bf83112d5894375762122ee4ec3d"
server_key_iv: "f8e1e77e7cf8796698131f84e54394de"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.578773 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "068c5f4a63314a63bf3cd99324b4cfcb"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "c457bf83112d5894375762122ee4ec3d"
server_key_iv: "f8e1e77e7cf8796698131f84e54394de"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.580044 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:49.580762 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:49.580933 5815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:49.581030 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "068c5f4a63314a63bf3cd99324b4cfcb"
format_stamp: "Formatted at 2026-06-19 20:49:49 on dist-test-slave-0smx"
server_key: "c457bf83112d5894375762122ee4ec3d"
server_key_iv: "f8e1e77e7cf8796698131f84e54394de"
server_key_version: "encryptionkey@0"
I20260619 20:49:49.581115 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:49.590032 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:49.590301 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:49.590443 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:49.594134 5815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42813
I20260619 20:49:49.594162 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42813 every 8 connection(s)
I20260619 20:49:49.594475 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:49.595357 5882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:49.597481 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb: Bootstrap starting.
I20260619 20:49:49.597967 5882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:49.598192 5882 log.cc:824] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:49.598819 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb: No bootstrap required, opened a new log
I20260619 20:49:49.600337 5882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42813 } }
I20260619 20:49:49.600461 5882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:49.600482 5882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068c5f4a63314a63bf3cd99324b4cfcb, State: Initialized, Role: FOLLOWER
I20260619 20:49:49.600603 5882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42813 } }
I20260619 20:49:49.600682 5882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:49.600703 5882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:49.600735 5882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:49.601191 5882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42813 } }
I20260619 20:49:49.601267 5882 leader_election.cc:302] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 068c5f4a63314a63bf3cd99324b4cfcb; no voters:
I20260619 20:49:49.601454 5882 leader_election.cc:288] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:49.601526 5885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:49.601697 5885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [term 1 LEADER]: Becoming Leader. State: Replica: 068c5f4a63314a63bf3cd99324b4cfcb, State: Running, Role: LEADER
I20260619 20:49:49.601717 5882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:49.601940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5815
I20260619 20:49:49.602061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:49.602253 392 external_mini_cluster.cc:1428] Setting key ee7d95a93b0772be1d7d483804cec617
I20260619 20:49:49.602366 5885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42813 } }
I20260619 20:49:49.602986 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42813 } } }
I20260619 20:49:49.602990 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 068c5f4a63314a63bf3cd99324b4cfcb. Latest consensus state: current_term: 1 leader_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068c5f4a63314a63bf3cd99324b4cfcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42813 } } }
I20260619 20:49:49.603073 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:49.603099 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:49.603448 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:49.603961 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:49.605625 5896 catalog_manager.cc:1347] Generated new cluster ID: b41d59ad1724497d87acb6a38d96b077
I20260619 20:49:49.605677 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:49.608564 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:49.602637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.0.98.62:42813)
0619 20:49:49.602900 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:49.602909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:49.603350 (+ 441us) server_negotiation.cc:407] Connection header received
0619 20:49:49.603475 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:49.603484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:49.603563 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:49.603696 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:49.603997 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.605373 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.606469 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.607093 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.607234 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.607896 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:49.607900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:49.607900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:49:49.608095 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:49.608135 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:49:49.608149 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:49.608227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:49.608302 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 20:49:49.608376 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20260619 20:49:49.609534 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:49:49.634466 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:49.635015 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:49.641008 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 068c5f4a63314a63bf3cd99324b4cfcb: Generated new TSK 0
I20260619 20:49:49.641193 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:49:49.703634 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:49.652853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.0.98.62:42813)
0619 20:49:49.652910 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:49.652912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:49.652995 (+ 83us) server_negotiation.cc:407] Connection header received
0619 20:49:49.653058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:49.653059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:49.653072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:49.653140 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:49:49.653248 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.653988 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.654704 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.655013 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.655105 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.655255 (+ 150us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:49:49.703338 (+ 48083us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 20:49:49.703551 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54270: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 20:49:49.703719 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54270: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 20:49:49.703684 5911 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:49.652779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42813 (local address 127.0.0.1:54270)
0619 20:49:49.652922 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:49.652932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:49.652993 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:49.653145 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:49.653146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:49.653147 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:49:49.653194 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:49.653195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.654059 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.654059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:49.654605 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:49.654606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:49.655100 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:49.655103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:49.655151 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:49.655155 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:49:49.703486 (+ 48331us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 20:49:49.703604 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
I20260619 20:49:49.704186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5815
2026-06-19T20:49:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (240 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:49 dist-test-slave-0smx krb5kdc[5923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:49 dist-test-slave-0smx krb5kdc[5923](info): set up 2 sockets
Jun 19 20:49:49 dist-test-slave-0smx krb5kdc[5923](info): commencing operation
krb5kdc: starting...
W20260619 20:49:50.913523 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:50 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:49:51.027863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:51.106500 5936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:51.106694 5936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:51.106717 5936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:51.107777 5936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:51.107811 5936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:51.107826 5936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:51.107838 5936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:51.107851 5936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:51.109265 5936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:51.109463 5936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:51.109627 5936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:51.114711 5942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.114775 5936 server_base.cc:1034] running on GCE node
W20260619 20:49:51.114705 5941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:51.114683 5944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.115038 5936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:51.115302 5936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:51.116444 5936 hybrid_clock.cc:648] HybridClock initialized: now 1781902191116433 us; error 29 us; skew 500 ppm
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.118822 5936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:49:51.119241 5936 webserver.cc:460] Webserver started at http://127.0.98.62:46711/ using document root <none> and password file <none>
I20260619 20:49:51.119427 5936 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:51.119467 5936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:51.119550 5936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:51.120527 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "866829d8ae61458e94cbe75cf3e4c1fb"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "06b0cf4e7f97ae082a5ea1f3088c1453"
server_key_iv: "5edf8be8dceb04dcb6c024c5271eb575"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.120824 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "866829d8ae61458e94cbe75cf3e4c1fb"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "06b0cf4e7f97ae082a5ea1f3088c1453"
server_key_iv: "5edf8be8dceb04dcb6c024c5271eb575"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.122128 5936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:49:51.122814 5951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.122983 5936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:51.123052 5936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "866829d8ae61458e94cbe75cf3e4c1fb"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "06b0cf4e7f97ae082a5ea1f3088c1453"
server_key_iv: "5edf8be8dceb04dcb6c024c5271eb575"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.123133 5936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:51.178823 5936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:51.180986 5936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:51.181134 5936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:51.184855 5936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38879
I20260619 20:49:51.184911 6003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38879 every 8 connection(s)
I20260619 20:49:51.185405 5936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:51.186106 6004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:51.188473 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb: Bootstrap starting.
I20260619 20:49:51.189119 6004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:51.189494 6004 log.cc:824] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:51.190177 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb: No bootstrap required, opened a new log
I20260619 20:49:51.191778 6004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38879 } }
I20260619 20:49:51.191920 6004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:51.191999 6004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866829d8ae61458e94cbe75cf3e4c1fb, State: Initialized, Role: FOLLOWER
I20260619 20:49:51.192179 6004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38879 } }
I20260619 20:49:51.192276 6004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:51.192329 6004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:51.192382 6004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:51.192837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 5936
I20260619 20:49:51.192926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:51.193104 392 external_mini_cluster.cc:1428] Setting key 2c9ae56455bd842200748bd922a63e79
I20260619 20:49:51.193538 6004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38879 } }
I20260619 20:49:51.193717 6004 leader_election.cc:302] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 866829d8ae61458e94cbe75cf3e4c1fb; no voters:
I20260619 20:49:51.193903 6004 leader_election.cc:288] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:51.193948 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:51.194114 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [term 1 LEADER]: Becoming Leader. State: Replica: 866829d8ae61458e94cbe75cf3e4c1fb, State: Running, Role: LEADER
I20260619 20:49:51.194207 6004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:51.194233 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38879 } }
I20260619 20:49:51.194638 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38879 } } }
I20260619 20:49:51.194729 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:51.194936 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 866829d8ae61458e94cbe75cf3e4c1fb. Latest consensus state: current_term: 1 leader_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "866829d8ae61458e94cbe75cf3e4c1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38879 } } }
I20260619 20:49:51.195173 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:51.195309 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:49:51.195796 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:51.197723 6015 catalog_manager.cc:1347] Generated new cluster ID: 7f824d71572f4d13b3b20ed2bd4da470
I20260619 20:49:51.197790 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:51.202867 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:51.203552 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:51.205240 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.193677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.0.98.62:38879)
0619 20:49:51.193903 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.193912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.193948 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:49:51.194073 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.194080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.194135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.195225 (+ 1090us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.195338 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.196886 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.197974 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.198613 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.198730 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.201553 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.201557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.201558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.201568 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.203708 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.204078 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.204080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.204081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.204180 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.204387 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.204388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.204388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.204630 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.204786 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.204881 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.205020 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260619 20:49:51.217015 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 866829d8ae61458e94cbe75cf3e4c1fb: Generated new TSK 0
I20260619 20:49:51.217192 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:49:51.252303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38879
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:51.332592 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:51.332800 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:51.332855 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:51.334005 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:51.334064 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:51.334110 6030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:51.335544 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:51.335790 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:51.336010 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:51.340971 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:51.340972 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.341060 6030 server_base.cc:1034] running on GCE node
W20260619 20:49:51.340972 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.341480 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:51.341706 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:51.342846 6030 hybrid_clock.cc:648] HybridClock initialized: now 1781902191342835 us; error 28 us; skew 500 ppm
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.345425 6030 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:49:51.345889 6030 webserver.cc:460] Webserver started at http://127.0.98.1:33605/ using document root <none> and password file <none>
I20260619 20:49:51.346140 6030 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:51.346211 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:51.346378 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:51.347420 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "8026f290133243a08a869008e653ca30"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "b1b5fc465f82f9a5cf389addade8b370"
server_key_iv: "67cbc4b9f76f201e2f4e412744837df7"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.347787 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8026f290133243a08a869008e653ca30"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "b1b5fc465f82f9a5cf389addade8b370"
server_key_iv: "67cbc4b9f76f201e2f4e412744837df7"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.349105 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:49:51.349907 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.350075 6030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.350171 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "8026f290133243a08a869008e653ca30"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "b1b5fc465f82f9a5cf389addade8b370"
server_key_iv: "67cbc4b9f76f201e2f4e412744837df7"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.350257 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:51.374627 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:51.376497 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:51.376665 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:51.376917 6030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:51.377216 6030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:51.377270 6030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.377329 6030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:51.377378 6030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.383133 6030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37737
I20260619 20:49:51.383183 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37737 every 8 connection(s)
I20260619 20:49:51.383481 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:51.387531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6030
I20260619 20:49:51.387667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:51.387905 392 external_mini_cluster.cc:1428] Setting key 9b9fd66c75a8d38fe512b0f787c2995a
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:51.393894 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.384126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38879 (local address 127.0.98.1:39833)
0619 20:49:51.384305 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:51.384341 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:51.385236 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:51.385512 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:51.385515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:51.385757 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:51.385939 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:51.385945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.386607 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.386608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:51.387396 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:51.387398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.387966 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.387967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:51.388140 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.388712 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:51.388726 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:51.390328 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:51.392508 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:51.392515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:51.392517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:51.392743 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:51.392972 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:51.392973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:51.392973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:51.393069 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:51.393408 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:51.393412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:51.393525 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 20:49:51.393609 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:49:51.393737 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260619 20:49:51.393914 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.384179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39833 (local address 127.0.98.62:38879)
0619 20:49:51.384261 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.384263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.384424 (+ 161us) server_negotiation.cc:407] Connection header received
0619 20:49:51.385388 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.385389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.385399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.385447 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.386016 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.386527 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.387482 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.387863 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.387938 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.390430 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.390432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.390433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.390442 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.392397 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.392827 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.392828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.392829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.392900 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.393140 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.393141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.393142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.393331 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.393422 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.393663 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.393765 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 20:49:51.394490 6159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38879
I20260619 20:49:51.394616 6159 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:51.394850 6159 heartbeater.cc:511] Master 127.0.98.62:38879 requested a full tablet report, sending...
I20260619 20:49:51.395480 5968 ts_manager.cc:194] Registered new tserver with Master: 8026f290133243a08a869008e653ca30 (127.0.98.1:37737)
I20260619 20:49:51.396582 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39833
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:49:51.414350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38879
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:51.494329 6164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:51.494577 6164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:51.494606 6164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:51.495749 6164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:51.495787 6164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:51.495812 6164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:51.497174 6164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:51.497397 6164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:51.497578 6164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:51.502377 6169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.502480 6164 server_base.cc:1034] running on GCE node
W20260619 20:49:51.502357 6172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:51.502362 6170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.502754 6164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:51.503024 6164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:51.504181 6164 hybrid_clock.cc:648] HybridClock initialized: now 1781902191504171 us; error 43 us; skew 500 ppm
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.507333 6164 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:49:51.507943 6164 webserver.cc:460] Webserver started at http://127.0.98.2:46605/ using document root <none> and password file <none>
I20260619 20:49:51.508285 6164 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:51.508347 6164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:51.508483 6164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:51.509970 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "ed0718fe9ed54edda3acdec234216f9d"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "0a91a4d5d83b3589282cfe35cfc05b39"
server_key_iv: "8713926d0bfe2ff49080b268406a8952"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.510382 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0718fe9ed54edda3acdec234216f9d"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "0a91a4d5d83b3589282cfe35cfc05b39"
server_key_iv: "8713926d0bfe2ff49080b268406a8952"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.512132 6164 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 20:49:51.513063 6179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.513263 6164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:51.513365 6164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "ed0718fe9ed54edda3acdec234216f9d"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "0a91a4d5d83b3589282cfe35cfc05b39"
server_key_iv: "8713926d0bfe2ff49080b268406a8952"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.513440 6164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:51.525071 6164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:51.526960 6164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:51.527093 6164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:51.527323 6164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:51.527614 6164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:51.527652 6164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.527675 6164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:51.527721 6164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.533232 6164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43015
I20260619 20:49:51.533342 6292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43015 every 8 connection(s)
I20260619 20:49:51.533623 6164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:51.538544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6164
I20260619 20:49:51.538655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:51.538856 392 external_mini_cluster.cc:1428] Setting key 20bb8efff2111fa30206d41fe5ea7113
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:51.542953 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.534303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47041 (local address 127.0.98.62:38879)
0619 20:49:51.534344 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.534346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.534623 (+ 277us) server_negotiation.cc:407] Connection header received
0619 20:49:51.535487 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.535488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.535497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.535549 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.536133 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.536638 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.537791 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.537967 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.538019 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.540115 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.540116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.540117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.540123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.541612 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.542028 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.542029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.542029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.542087 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.542343 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.542344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.542344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.542449 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.542516 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.542789 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.542861 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:51.543148 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.534281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38879 (local address 127.0.98.2:47041)
0619 20:49:51.534448 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:51.534489 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:51.535400 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:51.535610 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:51.535613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:51.535867 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:51.536046 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:51.536050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.536737 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.536738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:51.537719 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:51.537721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.538038 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.538039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:51.538135 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.538659 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:51.538669 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:51.540021 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:51.541732 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:51.541738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:51.541740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:51.541950 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:51.542165 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:51.542166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:51.542166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:51.542269 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:51.542520 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:51.542526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:51.542658 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 20:49:51.542757 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 20:49:51.542906 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260619 20:49:51.543617 6293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38879
I20260619 20:49:51.543717 6293 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:51.543882 6293 heartbeater.cc:511] Master 127.0.98.62:38879 requested a full tablet report, sending...
I20260619 20:49:51.544281 5968 ts_manager.cc:194] Registered new tserver with Master: ed0718fe9ed54edda3acdec234216f9d (127.0.98.2:43015)
I20260619 20:49:51.544661 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47041
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:49:51.562350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38879
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:51.640277 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:51.640462 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:51.640484 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:51.641549 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:51.641582 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:51.641606 6298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:51.642970 6298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:51.643138 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:51.643292 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:51.648058 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:51.648048 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:51.648036 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:51.648217 6298 server_base.cc:1034] running on GCE node
I20260619 20:49:51.648475 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:51.648664 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:51.649796 6298 hybrid_clock.cc:648] HybridClock initialized: now 1781902191649781 us; error 27 us; skew 500 ppm
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.651999 6298 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:49:51.652447 6298 webserver.cc:460] Webserver started at http://127.0.98.3:43579/ using document root <none> and password file <none>
I20260619 20:49:51.652668 6298 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:51.652736 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:51.652882 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:51.653995 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "650d8e3664a6498a9d2947fd05389424"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "d9ec96b80c4a37dcc2744615dd8a705f"
server_key_iv: "e4cb706f968d3888f224c294971591cf"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.654343 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "650d8e3664a6498a9d2947fd05389424"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "d9ec96b80c4a37dcc2744615dd8a705f"
server_key_iv: "e4cb706f968d3888f224c294971591cf"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.655570 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:49:51.656268 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.656432 6298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:51.656526 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "650d8e3664a6498a9d2947fd05389424"
format_stamp: "Formatted at 2026-06-19 20:49:51 on dist-test-slave-0smx"
server_key: "d9ec96b80c4a37dcc2744615dd8a705f"
server_key_iv: "e4cb706f968d3888f224c294971591cf"
server_key_version: "encryptionkey@0"
I20260619 20:49:51.656605 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:51.692006 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:51.693884 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:51.694072 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:51.694360 6298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:51.694696 6298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:51.694751 6298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.694813 6298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:51.694835 6298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:51.700848 6298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46527
I20260619 20:49:51.700894 6426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46527 every 8 connection(s)
I20260619 20:49:51.701192 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:51.706696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6298
I20260619 20:49:51.706785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:51.706943 392 external_mini_cluster.cc:1428] Setting key f3c6bc9226601df6e85e6c3ff7a05a75
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:49:51.710419 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.701906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42829 (local address 127.0.98.62:38879)
0619 20:49:51.701951 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.701954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.702190 (+ 236us) server_negotiation.cc:407] Connection header received
0619 20:49:51.703081 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.703082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.703091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.703147 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.703840 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.704307 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.705225 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.705556 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.705629 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.707829 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.707831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.707831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.707839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.709060 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.709518 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.709519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.709520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.709574 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.709818 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.709819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.709819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.709926 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.709995 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.710263 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.710336 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:51.710598 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.701904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38879 (local address 127.0.98.3:42829)
0619 20:49:51.702065 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:51.702099 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:51.702988 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:51.703308 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:51.703311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:51.703569 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:51.703757 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:51.703762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.704380 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.704381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:51.705146 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:51.705148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.705649 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.705650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:51.705745 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.706150 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:51.706157 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:51.707750 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:51.709141 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:51.709147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:51.709149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:51.709436 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:51.709642 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:51.709643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:51.709643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:51.709752 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:51.709997 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:51.710004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:51.710132 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 20:49:51.710230 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 20:49:51.710373 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260619 20:49:51.711045 6427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38879
I20260619 20:49:51.711118 6427 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:51.711302 6427 heartbeater.cc:511] Master 127.0.98.62:38879 requested a full tablet report, sending...
I20260619 20:49:51.711694 5968 ts_manager.cc:194] Registered new tserver with Master: 650d8e3664a6498a9d2947fd05389424 (127.0.98.3:46527)
I20260619 20:49:51.712086 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42829
I20260619 20:49:51.718950 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/test-admin.keytab.
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.733548 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 20:49:51.735137 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:51.734364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.0.98.1:37737)
0619 20:49:51.734553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.734557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.734566 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:51.734589 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.734590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.734602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.734670 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.735042 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54312: BlockingRecv error: recv got EOF from 127.0.0.1:54312 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
W20260619 20:49:51.735203 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.736155 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.737001 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:51.744339 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.737535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.0.98.1:37737)
0619 20:49:51.737596 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.737598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.737669 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:49:51.737705 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.737706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.737714 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.737764 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.738114 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.738810 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.739471 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.739762 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.739815 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.741770 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.741777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.741777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.741783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.743315 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.743638 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.743641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.743642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.743708 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.743863 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.743864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.743864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.743995 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.744106 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.744167 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.744249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 20:49:51.746033 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:51.745639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.0.98.1:37737)
0619 20:49:51.745690 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.745692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.745699 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:51.745718 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.745719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.745727 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.745784 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.746001 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54320: BlockingRecv error: recv got EOF from 127.0.0.1:54320 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 20:49:51.746137 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.747067 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.747881 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:51.754215 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.748378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.1:37737)
0619 20:49:51.748424 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.748426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.748434 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:51.748499 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.748499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.748506 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.748552 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.748873 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.749394 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.750049 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.750234 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.750284 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.752057 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.752058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.752059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.752064 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.753352 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.753663 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.753664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.753664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.753713 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.753871 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.753872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.753872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.753954 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.754020 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.754105 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.754160 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 20:49:51.755575 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:49:51.755117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.0.98.1:37737)
0619 20:49:51.755147 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.755149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.755181 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:49:51.755230 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.755230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.755237 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.755282 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.755491 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54334: BlockingRecv error: recv got EOF from 127.0.0.1:54334 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 20:49:51.755666 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781902166366061-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.756541 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:51.757408 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:49:51.764022 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:51.757855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.0.98.1:37737)
0619 20:49:51.757907 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:51.757909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:51.757972 (+ 63us) server_negotiation.cc:407] Connection header received
0619 20:49:51.758057 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:51.758057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:51.758066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:51.758137 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:51.758494 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.759009 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.759694 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:51.759882 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:51.759965 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:51.761890 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:51.761891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:51.761891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:51.761898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:51.762962 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.763285 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.763286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.763286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.763341 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:51.763550 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:51.763551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:51.763551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:51.763658 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:51.763754 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:51.763862 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:49:51.763932 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:49:51.764612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6030
I20260619 20:49:51.769007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6164
I20260619 20:49:51.773381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6298
I20260619 20:49:51.777427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 5936
2026-06-19T20:49:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1904 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[6447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[6447](info): set up 2 sockets
Jun 19 20:49:51 dist-test-slave-0smx krb5kdc[6447](info): commencing operation
krb5kdc: starting...
W20260619 20:49:52.863476 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 20:49:53.025132 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.162s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:49:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 20:49:53.132087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38413
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:53.213932 6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:53.214183 6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:53.214222 6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:53.215370 6460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:53.215446 6460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:53.215474 6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:53.215499 6460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:53.215530 6460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:53.216991 6460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:53.217264 6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:53.217541 6460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:53.222437 6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.222412 6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.222477 6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.222620 6460 server_base.cc:1034] running on GCE node
I20260619 20:49:53.222942 6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:53.223250 6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:53.224424 6460 hybrid_clock.cc:648] HybridClock initialized: now 1781902193224417 us; error 34 us; skew 500 ppm
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:53.229745 6460 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 20:49:53.230229 6460 webserver.cc:460] Webserver started at http://127.0.98.62:40301/ using document root <none> and password file <none>
I20260619 20:49:53.230456 6460 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.230525 6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.230659 6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:53.231725 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "26d99f894e3242b48d8e02d1ee115082"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "2185c9702bb36602f61ed626085aba49"
server_key_iv: "b8f6dfdee922211777cd397efd44cd69"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.232096 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "26d99f894e3242b48d8e02d1ee115082"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "2185c9702bb36602f61ed626085aba49"
server_key_iv: "b8f6dfdee922211777cd397efd44cd69"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.233472 6460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:49:53.234189 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.234359 6460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:53.234468 6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "26d99f894e3242b48d8e02d1ee115082"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "2185c9702bb36602f61ed626085aba49"
server_key_iv: "b8f6dfdee922211777cd397efd44cd69"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.234549 6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.246745 6460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:53.248847 6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:53.249040 6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:53.253201 6460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38413
I20260619 20:49:53.253233 6527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38413 every 8 connection(s)
I20260619 20:49:53.253610 6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:49:53.254597 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:53.256508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6460
I20260619 20:49:53.256604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:49:53.256827 392 external_mini_cluster.cc:1428] Setting key 0bafe35a01994c28dc34fc0c22709063
I20260619 20:49:53.257053 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Bootstrap starting.
I20260619 20:49:53.257771 6528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:53.258078 6528 log.cc:824] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:53.258751 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: No bootstrap required, opened a new log
I20260619 20:49:53.260480 6528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } }
I20260619 20:49:53.260634 6528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:53.260684 6528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d99f894e3242b48d8e02d1ee115082, State: Initialized, Role: FOLLOWER
I20260619 20:49:53.260807 6528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } }
I20260619 20:49:53.260881 6528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:49:53.260903 6528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:49:53.260962 6528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:53.261615 6528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } }
I20260619 20:49:53.261729 6528 leader_election.cc:302] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d99f894e3242b48d8e02d1ee115082; no voters:
I20260619 20:49:53.261916 6528 leader_election.cc:288] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:49:53.261945 6533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:53.262123 6533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 1 LEADER]: Becoming Leader. State: Replica: 26d99f894e3242b48d8e02d1ee115082, State: Running, Role: LEADER
I20260619 20:49:53.262172 6528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:53.262290 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } }
I20260619 20:49:53.262686 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d99f894e3242b48d8e02d1ee115082. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } }
I20260619 20:49:53.262950 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:53.262861 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } }
I20260619 20:49:53.263442 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:53.263562 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:49:53.264031 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:49:53.265666 6542 catalog_manager.cc:1347] Generated new cluster ID: 8f8c6cd2327344528a6100074b26c604
I20260619 20:49:53.265724 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:49:53.267602 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.257353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.98.62:38413)
0619 20:49:53.257545 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.257555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.257590 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:49:53.258004 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.258009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.258054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.258209 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.258852 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.259738 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.260477 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.261076 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.261202 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.264125 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.264129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.264129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.264142 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.266134 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.266511 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.266514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.266515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.266605 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.266810 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.266811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.266811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.267041 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.267209 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.267288 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.267448 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260619 20:49:53.280694 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:49:53.281487 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:49:53.288166 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Generated new TSK 0
I20260619 20:49:53.288357 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 20:49:53.318353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38413
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:53.397841 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:53.398038 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:53.398061 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:53.399187 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:53.399231 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:53.399257 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:53.400676 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:53.400902 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:53.401141 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:53.405961 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.405968 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.406054 6554 server_base.cc:1034] running on GCE node
W20260619 20:49:53.405969 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.406351 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:53.406594 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:53.407749 6554 hybrid_clock.cc:648] HybridClock initialized: now 1781902193407731 us; error 43 us; skew 500 ppm
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:53.410233 6554 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:49:53.410640 6554 webserver.cc:460] Webserver started at http://127.0.98.1:41521/ using document root <none> and password file <none>
I20260619 20:49:53.410858 6554 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.410926 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.411082 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:53.412092 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "191891f256bf49aa9ade310303d879b7"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "3f4d526ca0fa455039dd88cabbb41d23"
server_key_iv: "e738af2dfb3e670f46dcbdaaf9556b75"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.412464 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "191891f256bf49aa9ade310303d879b7"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "3f4d526ca0fa455039dd88cabbb41d23"
server_key_iv: "e738af2dfb3e670f46dcbdaaf9556b75"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.413795 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:53.414558 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.414762 6554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:49:53.414880 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "191891f256bf49aa9ade310303d879b7"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "3f4d526ca0fa455039dd88cabbb41d23"
server_key_iv: "e738af2dfb3e670f46dcbdaaf9556b75"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.414956 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.429557 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:53.431452 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:53.431602 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:53.431829 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:53.432128 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:53.432181 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.432224 6554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:53.432250 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.437927 6554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42179
I20260619 20:49:53.437978 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42179 every 8 connection(s)
I20260619 20:49:53.438330 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:49:53.442942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6554
I20260619 20:49:53.443070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:49:53.443269 392 external_mini_cluster.cc:1428] Setting key 156778468ad06f7a13f7a2e0919e3709
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:49:53.447742 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.438996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54441 (local address 127.0.98.62:38413)
0619 20:49:53.439060 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.439062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.439249 (+ 187us) server_negotiation.cc:407] Connection header received
0619 20:49:53.439715 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.439716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.439725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.439777 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.440657 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.441145 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.442107 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.442307 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.442382 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.444704 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.444705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.444706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.444714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.446286 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.446660 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.446661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.446662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.446726 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.446951 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.446952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.446952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.447106 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.447177 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.447510 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.447609 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:49:53.447786 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.438985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:54441)
0619 20:49:53.439127 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.439167 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.439634 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.439834 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.439837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.440405 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.440582 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.440586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.441225 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.441225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.442031 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.442033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.442409 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.442410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.442504 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.443072 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.443084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.444617 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.446356 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.446386 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.446387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.446586 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.446793 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.446794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.446794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.446886 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.447200 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.447205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.447320 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.447471 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.447632 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260619 20:49:53.448534 6683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38413
I20260619 20:49:53.448622 6683 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:53.448794 6683 heartbeater.cc:511] Master 127.0.98.62:38413 requested a full tablet report, sending...
I20260619 20:49:53.449323 6492 ts_manager.cc:194] Registered new tserver with Master: 191891f256bf49aa9ade310303d879b7 (127.0.98.1:42179)
I20260619 20:49:53.449932 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54441
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 20:49:53.466202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38413
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:53.545190 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:53.545442 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:53.545483 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:53.546633 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:53.546700 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:53.546746 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:53.548168 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:53.548419 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:53.548655 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:53.553552 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.553594 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.553560 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.553893 6688 server_base.cc:1034] running on GCE node
I20260619 20:49:53.554070 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:53.554260 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:53.555390 6688 hybrid_clock.cc:648] HybridClock initialized: now 1781902193555376 us; error 28 us; skew 500 ppm
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:53.557781 6688 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:49:53.558230 6688 webserver.cc:460] Webserver started at http://127.0.98.2:35275/ using document root <none> and password file <none>
I20260619 20:49:53.558467 6688 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.558555 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.558705 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:53.559705 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "1aea3357cc03b23441603b2e67d41d84"
server_key_iv: "5eb983499bcdd67126feafd6481c43b5"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.560057 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "1aea3357cc03b23441603b2e67d41d84"
server_key_iv: "5eb983499bcdd67126feafd6481c43b5"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.561427 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:49:53.562111 6703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.562288 6688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:53.562353 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "1aea3357cc03b23441603b2e67d41d84"
server_key_iv: "5eb983499bcdd67126feafd6481c43b5"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.562444 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.569049 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:53.571185 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:53.571337 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:53.571547 6688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:53.571826 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:53.571858 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.571878 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:53.571930 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.577247 6688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37323
I20260619 20:49:53.577338 6816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37323 every 8 connection(s)
I20260619 20:49:53.577673 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:49:53.580863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6688
I20260619 20:49:53.580950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:49:53.581157 392 external_mini_cluster.cc:1428] Setting key 30c0197de629981e6b4a11044dfe37ae
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:49:53.587651 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.578416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35589 (local address 127.0.98.62:38413)
0619 20:49:53.578456 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.578458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.578662 (+ 204us) server_negotiation.cc:407] Connection header received
0619 20:49:53.579186 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.579188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.579200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.579252 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.580162 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.580704 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.582102 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.582421 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.582508 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.584722 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.584724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.584724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.584730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.586199 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.586726 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.586726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.586727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.586775 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.587058 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.587059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.587059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.587153 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.587204 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.587529 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.587588 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:53.587935 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.578392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:35589)
0619 20:49:53.578537 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.578590 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.579060 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.579314 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.579319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.579875 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.580063 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.580067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.580800 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.580801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.582025 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.582027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.582505 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.582505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.582598 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.583153 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.583164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.584637 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.586280 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.586287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.586289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.586649 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.586840 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.586841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.586842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.586986 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.587217 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.587222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.587386 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.587518 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.587693 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260619 20:49:53.588429 6817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38413
I20260619 20:49:53.588544 6817 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:53.588739 6817 heartbeater.cc:511] Master 127.0.98.62:38413 requested a full tablet report, sending...
I20260619 20:49:53.589108 6492 ts_manager.cc:194] Registered new tserver with Master: 4789f58f8b4d4ab5a7fdebeb8c70b5a8 (127.0.98.2:37323)
I20260619 20:49:53.589545 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35589
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 20:49:53.605540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38413
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:53.684001 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:53.684219 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:53.684257 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:53.685433 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:53.685496 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:53.685540 6822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:53.686942 6822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:53.687201 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:53.687443 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:53.692287 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.692314 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.692265 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.692421 6822 server_base.cc:1034] running on GCE node
I20260619 20:49:53.692615 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:53.692806 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:53.693938 6822 hybrid_clock.cc:648] HybridClock initialized: now 1781902193693925 us; error 28 us; skew 500 ppm
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:53.696380 6822 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:49:53.696802 6822 webserver.cc:460] Webserver started at http://127.0.98.3:43897/ using document root <none> and password file <none>
I20260619 20:49:53.696973 6822 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.697026 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.697109 6822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:53.698163 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c95658cbb6a4fdda85cfdac402e00b1"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "cc78196167d1cb3a9c03df66325b0be9"
server_key_iv: "8f4e8c92f1b65fe9c8ab1a80006c2725"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.698453 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c95658cbb6a4fdda85cfdac402e00b1"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "cc78196167d1cb3a9c03df66325b0be9"
server_key_iv: "8f4e8c92f1b65fe9c8ab1a80006c2725"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.699613 6822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:49:53.700317 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.700512 6822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:53.700582 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "9c95658cbb6a4fdda85cfdac402e00b1"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "cc78196167d1cb3a9c03df66325b0be9"
server_key_iv: "8f4e8c92f1b65fe9c8ab1a80006c2725"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.700672 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.729522 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:53.731429 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:53.731591 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:53.731824 6822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:53.732136 6822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:53.732189 6822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.732224 6822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:53.732251 6822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.738029 6822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34371
I20260619 20:49:53.738070 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34371 every 8 connection(s)
I20260619 20:49:53.738377 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:49:53.739858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6822
I20260619 20:49:53.739970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:49:53.740139 392 external_mini_cluster.cc:1428] Setting key e652334b4dfbe110b629f54c187121c3
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:49:53.747634 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.739090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40819 (local address 127.0.98.62:38413)
0619 20:49:53.739166 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.739168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.739379 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:49:53.739888 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.739890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.739902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.739962 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.741176 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.741741 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.742742 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.742952 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.743045 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.745009 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.745010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.745010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.745017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.746242 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.746687 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.746689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.746689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.746751 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.747002 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.747003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.747003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.747116 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.747193 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.747459 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.747537 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 20:49:53.747828 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.739093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:40819)
0619 20:49:53.739257 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.739292 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.739781 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.740034 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.740038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.740807 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.741083 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.741095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.741835 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.741836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.742636 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.742637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.743053 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.743055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.743184 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.743608 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.743615 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.744899 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.746339 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.746346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.746348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.746606 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.746819 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.746820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.746820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.746931 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.747185 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.747191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.747329 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.747438 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.747601 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260619 20:49:53.748294 6951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38413
I20260619 20:49:53.748385 6951 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:53.748579 6951 heartbeater.cc:511] Master 127.0.98.62:38413 requested a full tablet report, sending...
I20260619 20:49:53.748940 6492 ts_manager.cc:194] Registered new tserver with Master: 9c95658cbb6a4fdda85cfdac402e00b1 (127.0.98.3:34371)
I20260619 20:49:53.749277 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40819
I20260619 20:49:53.751924 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 20:49:53.775866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42575
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38413,127.0.98.61:42575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:53.855182 6956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:53.855377 6956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:53.855398 6956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:53.856509 6956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:53.856549 6956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:53.856563 6956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:53.856575 6956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:53.856640 6956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:53.858155 6956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38413,127.0.98.61:42575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
W20260619 20:49:53.858250 6956 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38413,127.0.98.61:42575'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 20:49:53.858415 6956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:53.858647 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:53.863258 6962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.863319 6961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.863253 6964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.863453 6956 server_base.cc:1034] running on GCE node
I20260619 20:49:53.863677 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:53.864003 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:53.865178 6956 hybrid_clock.cc:648] HybridClock initialized: now 1781902193865169 us; error 26 us; skew 500 ppm
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:53.867587 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 20:49:53.868044 6956 webserver.cc:460] Webserver started at http://127.0.98.61:35559/ using document root <none> and password file <none>
I20260619 20:49:53.868258 6956 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.868300 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.868448 6956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:53.869546 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/instance:
uuid: "0c93f80caa404069886909d51fc4df50"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "d0193dbfcf8798e64ce483a42b44724f"
server_key_iv: "c1874a08f91ec320d0bc43b3aed612b2"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.869894 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal/instance:
uuid: "0c93f80caa404069886909d51fc4df50"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "d0193dbfcf8798e64ce483a42b44724f"
server_key_iv: "c1874a08f91ec320d0bc43b3aed612b2"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.871126 6956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:53.871784 6971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.871963 6956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.872041 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
uuid: "0c93f80caa404069886909d51fc4df50"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "d0193dbfcf8798e64ce483a42b44724f"
server_key_iv: "c1874a08f91ec320d0bc43b3aed612b2"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.872088 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.880599 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:53.882730 6956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:53.882833 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:53.886543 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42575
I20260619 20:49:53.886621 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42575 every 8 connection(s)
I20260619 20:49:53.886924 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:49:53.887936 7024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:53.889501 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:49:53.890115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 6956
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:49:53.899343 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.890212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.0.98.62:38413)
0619 20:49:53.890269 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.890272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.890542 (+ 270us) server_negotiation.cc:407] Connection header received
0619 20:49:53.891226 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.891227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.891240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.891307 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.891817 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.892319 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.893379 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.893679 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.893729 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.896156 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.896158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.896158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.896166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.897973 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.898387 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.898388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.898389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.898455 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.898688 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.898689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.898689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.898836 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.898909 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.899178 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.899254 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:49:53.899515 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.890203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:38176)
0619 20:49:53.890387 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.890434 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.891108 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.891341 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.891344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.891586 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.891725 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.891729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.892408 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.892409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.893272 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.893274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.893759 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.893760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.893907 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.894463 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.894472 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.896068 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.898065 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.898071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.898073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.898302 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.898521 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.898522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.898523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.898622 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.898914 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.898920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.899045 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.899140 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.899282 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260619 20:49:53.899973 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.890701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.98.61:42575)
0619 20:49:53.890995 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.890999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.891030 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:49:53.891110 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.891117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.891138 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.891222 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.891857 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.892821 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.893665 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.894101 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.894209 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.896741 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.896749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.896749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.896758 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.898490 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.898867 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.898870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.898870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.898929 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.899131 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.899132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.899133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.899264 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.899751 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.899801 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.899873 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":79,"threads_started":1}
I20260619 20:49:53.900251 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:49:53.905750 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.900467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.0.1:52260)
0619 20:49:53.900513 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.900522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.900557 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.900679 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.900679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.900798 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.900840 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.900841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.901467 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.901468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.901922 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.901923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.902220 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.902220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.902267 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.902644 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.902649 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.903677 (+ 1028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.904934 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.904935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.904935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.905086 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.905242 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.905242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.905243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.905287 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.905529 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.905530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.905560 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.905610 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.905676 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:49:53.905797 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.900523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.0.98.61:42575)
0619 20:49:53.900556 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.900558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.900566 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:53.900608 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.900609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.900618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.900664 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.900895 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.901401 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.901990 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.902164 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.902209 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.903744 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.903745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.903745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.903751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.904868 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.905145 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.905146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.905146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.905188 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.905395 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.905396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.905396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.905480 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.905523 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.905623 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.905714 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":18}
I20260619 20:49:53.907070 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Bootstrap starting.
I20260619 20:49:53.907850 7024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:53.908128 7024 log.cc:824] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:53.908788 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: No bootstrap required, opened a new log
I20260619 20:49:53.910179 7024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } }
I20260619 20:49:53.910295 7024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:53.910337 7024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c93f80caa404069886909d51fc4df50, State: Initialized, Role: FOLLOWER
I20260619 20:49:53.910461 7024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } }
I20260619 20:49:53.910684 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } } }
I20260619 20:49:53.910733 7024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:53.910771 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:49:53.912370 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:49:53.912447 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:49:53.913940 6956 master_runner.cc:428] Detected that this master 0c93f80caa404069886909d51fc4df50 is joining an existing cluster
I20260619 20:49:53.914002 6956 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:42575
I20260619 20:49:53.921819 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.914864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:38192)
0619 20:49:53.915182 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.915193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.915257 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.915440 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.915441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.915683 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.915755 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.915757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.916633 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.916634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.917362 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.917364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.917906 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.917907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.917971 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.918575 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.918579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.918954 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.920646 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.920647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.920647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.920899 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.921109 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.921110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.921110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.921165 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.921451 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.921452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.921493 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.921589 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.921685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260619 20:49:53.921979 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.914933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.0.98.62:38413)
0619 20:49:53.914976 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.914978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.915259 (+ 281us) server_negotiation.cc:407] Connection header received
0619 20:49:53.915319 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.915320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.915329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.915396 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.915854 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.916569 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.917504 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.917790 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.917832 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.919039 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.919040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.919041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.919048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.920549 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.920987 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.920988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.920989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.921042 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.921236 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.921237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.921237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.921374 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.921454 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.921815 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.921892 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:49:53.922089 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.915126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.61:42575)
0619 20:49:53.915157 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.915158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.915281 (+ 123us) server_negotiation.cc:407] Connection header received
0619 20:49:53.915367 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.915368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.915375 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.915425 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.915794 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.916500 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.917541 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.917807 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.917870 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.919082 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.919083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.919084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.919089 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.920960 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.921333 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.921334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.921334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.921391 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.921584 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.921585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.921585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.921702 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.921766 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.921935 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.922006 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:49:53.922497 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.915093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.0.1:52276)
0619 20:49:53.915229 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.915237 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.915299 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.915430 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.915430 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.915637 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.915717 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.915718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.916581 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.916582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.917443 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.917446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.917891 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.917892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.917957 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.918497 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.918501 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.918994 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.921041 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.921041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.921042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.921230 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.921455 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.921455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.921456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.921519 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.921774 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.921775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.921816 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.922153 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.922232 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:49:53.930998 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.924540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.98.61:42575)
0619 20:49:53.924575 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.924578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.924683 (+ 105us) server_negotiation.cc:407] Connection header received
0619 20:49:53.924767 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.924768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.924777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.924831 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.925250 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.925720 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.926465 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.926662 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.926743 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.928673 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.928674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.928674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.928680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.929961 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.930359 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.930359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.930359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.930402 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.930584 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.930585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.930585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.930669 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.930715 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.930894 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.930945 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:53.931094 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.924471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.0.1:52286)
0619 20:49:53.924584 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.924623 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.924705 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.924837 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.924840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.925127 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.925168 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.925170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.925799 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.925800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.926382 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.926383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.926745 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.926746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.926817 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.927257 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.927261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.928583 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.930033 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.930039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.930041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.930290 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.930462 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.930463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.930463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.930527 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.930730 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.930735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.930788 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.930874 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.930991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
I20260619 20:49:53.931643 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:42575: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0c93f80caa404069886909d51fc4df50"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 42575
}
attrs {
promote: true
}
}
dest_uuid: "26d99f894e3242b48d8e02d1ee115082"
cas_config_opid_index: -1
I20260619 20:49:53.931933 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } }
I20260619 20:49:53.932379 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c93f80caa404069886909d51fc4df50 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } }
I20260619 20:49:53.932605 6998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:53.933254 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:42575
I20260619 20:49:53.933472 6956 master.cc:537] Master@127.0.98.61:42575 shutting down...
I20260619 20:49:53.933547 6998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Refusing update from remote peer 26d99f894e3242b48d8e02d1ee115082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 20:49:53.933513 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } }
I20260619 20:49:53.933591 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:53.933853 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:49:53.934547 6477 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 20:49:53.935173 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 -> Peer 0c93f80caa404069886909d51fc4df50 (127.0.98.61:42575): Couldn't send request to peer 0c93f80caa404069886909d51fc4df50. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:49:53.936185 6956 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:53.936265 6956 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:53.936313 6956 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: stopping tablet replica
I20260619 20:49:53.948799 6956 master.cc:552] Master@127.0.98.61:42575 shutdown complete.
I20260619 20:49:53.948848 6956 master_runner.cc:305] Clearing existing system tablet
I20260619 20:49:53.948975 6956 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.949012 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.949729 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:53.950232 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.950353 6956 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.950410 6956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.950464 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
uuid: "0c93f80caa404069886909d51fc4df50"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "d0193dbfcf8798e64ce483a42b44724f"
server_key_iv: "c1874a08f91ec320d0bc43b3aed612b2"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.951565 6956 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:53.952903 6956 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 20:49:53.952973 6956 log.cc:1196] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 20:49:53.953157 6956 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Deleting consensus metadata
I20260619 20:49:53.953274 6956 master_runner.cc:315] Copying system tablet from 127.0.98.62:38413
I20260619 20:49:53.953886 6956 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38413
I20260619 20:49:53.959740 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.954202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.0.98.62:38413)
0619 20:49:53.954241 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:53.954243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:53.954334 (+ 91us) server_negotiation.cc:407] Connection header received
0619 20:49:53.954398 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:53.954399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:53.954408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:53.954479 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:53.954798 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.955277 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.956017 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.956200 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.956300 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.957215 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:53.957216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:53.957217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:53.957224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:53.958645 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.959007 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.959008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.959009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.959066 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:53.959255 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:53.959256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:53.959256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:53.959364 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:53.959437 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:53.959584 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:49:53.959654 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:53.959755 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:53.954154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:38208)
0619 20:49:53.954264 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:53.954273 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:53.954335 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:53.954482 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:53.954483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:53.954646 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:53.954706 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.954707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.955357 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.955358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.955919 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:53.955921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:53.956299 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:53.956300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:53.956364 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:53.956823 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:53.956828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:53.957110 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:53.958746 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.958747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.958747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.958926 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.959137 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:53.959138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:53.959138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:53.959197 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:53.959427 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:53.959428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:53.959477 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:49:53.959559 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:49:53.959656 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260619 20:49:53.960248 6512 tablet_copy_service.cc:140] P 26d99f894e3242b48d8e02d1ee115082: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0c93f80caa404069886909d51fc4df50 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38208)
I20260619 20:49:53.960348 6512 tablet_copy_service.cc:161] P 26d99f894e3242b48d8e02d1ee115082: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0c93f80caa404069886909d51fc4df50 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38208: session id = 0c93f80caa404069886909d51fc4df50-00000000000000000000000000000000
I20260619 20:49:53.961118 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 20:49:53.961756 6956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 20:49:53.962749 6956 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: tablet copy: Starting download of 0 data blocks...
I20260619 20:49:53.962896 6956 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: tablet copy: Starting download of 1 WAL segments...
I20260619 20:49:53.963748 6956 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 20:49:53.964520 6512 tablet_copy_service.cc:342] P 26d99f894e3242b48d8e02d1ee115082: Request end of tablet copy session 0c93f80caa404069886909d51fc4df50-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38208
I20260619 20:49:53.964574 6512 tablet_copy_service.cc:433] P 26d99f894e3242b48d8e02d1ee115082: ending tablet copy session 0c93f80caa404069886909d51fc4df50-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0c93f80caa404069886909d51fc4df50
I20260619 20:49:53.966992 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:53.968044 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.968055 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:53.968058 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:53.968338 6956 server_base.cc:1034] running on GCE node
I20260619 20:49:53.968461 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:53.968596 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:53.969712 6956 hybrid_clock.cc:648] HybridClock initialized: now 1781902193969705 us; error 23 us; skew 500 ppm
Jun 19 20:49:53 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:53.971055 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 20:49:53.971324 6956 webserver.cc:460] Webserver started at http://127.0.98.61:45565/ using document root <none> and password file <none>
I20260619 20:49:53.971460 6956 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:53.971515 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:53.972074 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.972579 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.972723 6956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:53.972803 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
uuid: "0c93f80caa404069886909d51fc4df50"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "d0193dbfcf8798e64ce483a42b44724f"
server_key_iv: "c1874a08f91ec320d0bc43b3aed612b2"
server_key_version: "encryptionkey@0"
I20260619 20:49:53.972857 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:53.999148 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:54.000869 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:54.003998 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42575
I20260619 20:49:54.004014 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42575 every 8 connection(s)
I20260619 20:49:54.004307 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:49:54.004976 7125 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:49:54.005437 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Bootstrap starting.
I20260619 20:49:54.007542 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:49:54.007750 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: Bootstrap complete.
I20260619 20:49:54.007936 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } }
I20260619 20:49:54.008015 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c93f80caa404069886909d51fc4df50, State: Initialized, Role: LEARNER
I20260619 20:49:54.008061 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } }
I20260619 20:49:54.008225 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:54.008199 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } }
I20260619 20:49:54.008265 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:49:54.009547 7139 catalog_manager.cc:1259] Loaded cluster ID: 8f8c6cd2327344528a6100074b26c604
I20260619 20:49:54.009593 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: loading cluster ID for follower catalog manager: success
I20260619 20:49:54.010536 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: acquiring CA information for follower catalog manager: success
I20260619 20:49:54.010818 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:49:54.061199 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.052028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.98.61:42575)
0619 20:49:54.052361 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.052363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.052371 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:54.052402 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.052403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.052415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.053594 (+ 1179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.053636 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.054654 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.055776 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.056088 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.056415 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.057793 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.057794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.057794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.057801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.059529 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.059954 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.059955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.059956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.060017 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.060262 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.060264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.060264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.060403 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.060934 (+ 531us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.060979 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.061067 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 20:49:54.088642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43969
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:49:54.169502 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:54.169704 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:54.169725 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:54.170862 7143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:49:54.170905 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:54.170920 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:54.170971 7143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:49:54.170989 7143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:49:54.172518 7143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:54.172757 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:54.173007 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:49:54.177989 7143 server_base.cc:1034] running on GCE node
W20260619 20:49:54.178053 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.177991 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.178126 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:54.178357 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:54.178658 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:54.179823 7143 hybrid_clock.cc:648] HybridClock initialized: now 1781902194179816 us; error 36 us; skew 500 ppm
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:54.182463 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 20:49:54.182964 7143 webserver.cc:460] Webserver started at http://127.0.98.60:43127/ using document root <none> and password file <none>
I20260619 20:49:54.183189 7143 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:54.183269 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:54.183424 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:49:54.184518 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/instance:
uuid: "d985fb669d8849c8af87bea35fd6dce5"
format_stamp: "Formatted at 2026-06-19 20:49:54 on dist-test-slave-0smx"
server_key: "c8295597b7d404b26995c1feaa76b648"
server_key_iv: "9af11c9cf64402377727c8a224ee4ffd"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.184904 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal/instance:
uuid: "d985fb669d8849c8af87bea35fd6dce5"
format_stamp: "Formatted at 2026-06-19 20:49:54 on dist-test-slave-0smx"
server_key: "c8295597b7d404b26995c1feaa76b648"
server_key_iv: "9af11c9cf64402377727c8a224ee4ffd"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.186478 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:49:54.187261 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.187449 7143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:54.187556 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
uuid: "d985fb669d8849c8af87bea35fd6dce5"
format_stamp: "Formatted at 2026-06-19 20:49:54 on dist-test-slave-0smx"
server_key: "c8295597b7d404b26995c1feaa76b648"
server_key_iv: "9af11c9cf64402377727c8a224ee4ffd"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.187633 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:54.213739 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:54.216260 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:54.216437 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:54.221586 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43969
I20260619 20:49:54.221621 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43969 every 8 connection(s)
I20260619 20:49:54.221997 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:49:54.222939 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:54.223340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 7143
I20260619 20:49:54.224622 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:49:54.233175 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.225364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.98.62:38413)
0619 20:49:54.225408 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.225410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.225528 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:49:54.225621 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.225623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.225634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.225708 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.226175 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.226678 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.227504 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.227696 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.227771 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.229876 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.229877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.229878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.229884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.231685 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.232083 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.232085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.232085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.232148 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.232448 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.232449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.232449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.232596 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.232670 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.232987 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.233070 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:49:54.233367 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.225189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:38212)
0619 20:49:54.225408 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.225461 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.225546 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.225726 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.225729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.225984 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.226086 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.226090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.226781 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.226782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.227410 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.227418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.227777 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.227778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.227850 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.228327 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.228334 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.229791 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.231763 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.231769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.231770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.232009 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.232216 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.232217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.232217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.232365 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.232685 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.232691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.232857 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.232942 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.233103 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":20,"thread_start_us":153,"threads_started":1}
I20260619 20:49:54.233389 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.223940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.0.98.60:43969)
0619 20:49:54.224117 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.224124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.224147 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:49:54.224558 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.224564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.224609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.224750 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.225515 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.226394 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.227116 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.227511 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.227626 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.230455 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.230464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.230464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.230471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.232200 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.232565 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.232568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.232568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.232622 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.232810 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.232811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.232811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.232950 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.233086 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.233173 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.233246 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260619 20:49:54.234236 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:49:54.240063 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.234474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.0.1:52306)
0619 20:49:54.234530 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.234541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.234581 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.234695 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.234695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.234797 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.234840 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.234842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.235428 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.235429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.235894 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.235896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.236199 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.236200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.236245 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.236658 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.236662 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.237852 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.239309 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.239310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.239310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.239465 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.239628 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.239628 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.239629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.239673 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.239854 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.239855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.239885 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.239937 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.240002 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 20:49:54.240072 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.234549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.0.98.61:42575)
0619 20:49:54.234609 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.234611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.234618 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:54.234638 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.234638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.234649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.234694 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.234895 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.235359 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.235959 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.236141 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.236190 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.237947 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.237949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.237949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.237956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.239234 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.239526 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.239526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.239527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.239574 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.239709 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.239709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.239710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.239808 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.239852 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.239952 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.240012 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:49:54.240451 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43969 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:49:54.246532 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.240684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.0.1:55166)
0619 20:49:54.240728 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.240738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.240776 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.240882 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.240883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.241003 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.241026 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.241027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.241774 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.241775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.242262 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.242263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.242640 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.242641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.242686 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.243098 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.243103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.244172 (+ 1069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.245541 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.245542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.245542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.245697 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.245908 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.245909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.245909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.245961 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.246223 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.246224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.246259 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.246327 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.246423 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:49:54.246575 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.240745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.0.98.60:43969)
0619 20:49:54.240793 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.240795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.240802 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:54.240821 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.240822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.240830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.240871 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.241097 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.241689 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.242351 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.242561 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.242643 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.244261 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.244262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.244263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.244269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.245454 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.245789 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.245790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.245790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.245837 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.246053 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.246054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.246054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.246161 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.246233 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.246347 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.246469 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":33}
I20260619 20:49:54.248219 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Bootstrap starting.
I20260619 20:49:54.248796 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:54.249017 7211 log.cc:824] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:54.249730 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: No bootstrap required, opened a new log
I20260619 20:49:54.251142 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43969 } }
I20260619 20:49:54.251288 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:54.251338 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d985fb669d8849c8af87bea35fd6dce5, State: Initialized, Role: FOLLOWER
I20260619 20:49:54.251467 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43969 } }
I20260619 20:49:54.251718 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43969 } } }
I20260619 20:49:54.251797 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:54.251804 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:49:54.253484 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:49:54.253537 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:49:54.255371 7143 master_runner.cc:428] Detected that this master d985fb669d8849c8af87bea35fd6dce5 is joining an existing cluster
I20260619 20:49:54.255436 7143 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43969
I20260619 20:49:54.263062 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.256841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.98.61:42575)
0619 20:49:54.256884 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.256886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.256893 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:54.256912 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.256912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.256921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.256970 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.257383 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.258084 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259142 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.259439 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259474 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.260553 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.260555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.260555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.260562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.261964 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.262282 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.262283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.262284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.262336 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.262580 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.262581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.262581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.262686 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.262787 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.262906 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.262977 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:49:54.263252 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.256563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.0.1:52322)
0619 20:49:54.256654 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.256664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.256704 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.257016 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.257017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.257219 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.257270 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.257271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.258167 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.258168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.259046 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.259048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.259561 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.259625 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.260113 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.260118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.260440 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.262034 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.262034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.262035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.262200 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.262425 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.262426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.262427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.262482 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.262774 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.262775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.262820 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.262884 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.262978 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":38,"thread_start_us":53,"threads_started":1}
I20260619 20:49:54.284637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6554
I20260619 20:49:54.285590 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.256853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.0.98.60:43969)
0619 20:49:54.256887 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.256889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.256993 (+ 104us) server_negotiation.cc:407] Connection header received
0619 20:49:54.257074 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.257075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.257083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.257108 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.257543 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.258238 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259624 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.259898 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259935 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.282559 (+ 22624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.282560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.282560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.282566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.284333 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.284696 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.284697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.284697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.284740 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.285033 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.285034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.285034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.285167 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.285206 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.285418 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.285497 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:54.285676 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.256381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.62:38413)
0619 20:49:54.256421 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.256424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.256562 (+ 138us) server_negotiation.cc:407] Connection header received
0619 20:49:54.256648 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.256649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.256661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.256695 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.257182 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.258003 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259142 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.259430 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259472 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.282527 (+ 23055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.282529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.282529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.282538 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.284498 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.284860 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.284861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.284861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.284913 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.285112 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.285113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.285113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.285234 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.285273 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.285510 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.285579 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:49:54.285985 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.256316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:38220)
0619 20:49:54.256488 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.256506 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.256576 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.256772 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.256773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.257007 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.257094 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.257096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.258090 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.258091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.259046 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.259048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.259564 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.259625 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.281959 (+ 22334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.281964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.282406 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.284581 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.284582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.284582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.284780 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.284981 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.284981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.284982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.285042 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.285359 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.285360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.285398 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.285789 (+ 391us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.285898 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":39,"thread_start_us":126,"threads_started":1}
I20260619 20:49:54.286520 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.256703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.0.1:55168)
0619 20:49:54.256939 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.256947 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.257012 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.257159 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.257159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.257395 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.257460 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.257463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.258332 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.258333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.259533 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.259535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.259997 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.259997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.260054 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.281890 (+ 21836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.281897 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.282453 (+ 556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.284426 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.284427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.284428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.284624 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.284906 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.284906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.284906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.284966 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.285246 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.285246 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.285284 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.286321 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.286419 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":46,"threads_started":1}
I20260619 20:49:54.290926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42179
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38413
--webserver_port=41521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:49:54.298359 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.289445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.0.1:55176)
0619 20:49:54.290475 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.290486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.290556 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.290690 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.290691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.290838 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.290876 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.290878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.291701 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.291702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.292450 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.292452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.292918 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.292919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.292989 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.293402 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.293407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.294920 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.297175 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.297176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.297177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.297428 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.297640 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.297641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.297641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.297704 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.298036 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.298037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.298082 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.298156 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.298255 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012}
I20260619 20:49:54.298393 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.290428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.60:43969)
0619 20:49:54.290464 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.290466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.290540 (+ 74us) server_negotiation.cc:407] Connection header received
0619 20:49:54.290614 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.290615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.290625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.290686 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.290949 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.291614 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.292537 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.292834 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.292908 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.295404 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.295405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.295406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.295413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.297086 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.297505 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.297506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.297506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.297564 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.297810 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.297811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.297811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.297961 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.298037 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.298198 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.298277 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:49:54.299110 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43969: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43969
}
attrs {
promote: true
}
}
dest_uuid: "26d99f894e3242b48d8e02d1ee115082"
cas_config_opid_index: 5
I20260619 20:49:54.299342 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:49:54.299804 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d985fb669d8849c8af87bea35fd6dce5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.300740 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43969
I20260619 20:49:54.300812 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.300920 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:54.301332 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:54.302472 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 1 FOLLOWER]: Refusing update from remote peer 26d99f894e3242b48d8e02d1ee115082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 20:49:54.302906 7143 master.cc:537] Master@127.0.98.60:43969 shutting down...
I20260619 20:49:54.303045 7237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:49:54.303699 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 -> Peer d985fb669d8849c8af87bea35fd6dce5 (127.0.98.60:43969): Couldn't send request to peer d985fb669d8849c8af87bea35fd6dce5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:49:54.306779 7143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:54.306861 7143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:54.306902 7143 tablet_replica.cc:330] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: stopping tablet replica
I20260619 20:49:54.309036 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.301154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.0.98.61:42575)
0619 20:49:54.301204 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.301206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.303395 (+ 2189us) server_negotiation.cc:407] Connection header received
0619 20:49:54.303425 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.303426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.303449 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.303516 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:54.305683 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.306467 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.307912 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.308526 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.308570 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.308653 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.308904 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.308946 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:49:54.309281 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.301038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.0.1:52332)
0619 20:49:54.302861 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.302872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.302919 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.305550 (+ 2631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.305551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.305554 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:54.305591 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.305593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.306647 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.306648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.307804 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.307806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.308734 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.308735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.308800 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.308830 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.309144 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.309191 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809}
I20260619 20:49:54.309569 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 LEARNER]: Refusing update from remote peer 26d99f894e3242b48d8e02d1ee115082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 20:49:54.309767 7237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 20:49:54.310348 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d985fb669d8849c8af87bea35fd6dce5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.311544 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d99f894e3242b48d8e02d1ee115082. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.311636 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:49:54.311694 7051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: attempting to promote NON_VOTER 0c93f80caa404069886909d51fc4df50 to VOTER
I20260619 20:49:54.311676 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: true } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.311750 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:49:54.311861 7051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:49:54.312507 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.312394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.0.1:55188)
0619 20:49:54.312442 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.312483 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:49:54.312544 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 LEARNER]: Refusing update from remote peer 26d99f894e3242b48d8e02d1ee115082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 20:49:54.312839 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 20:49:54.313133 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.313040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.0.1:55190)
0619 20:49:54.313088 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.313114 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:49:54.313344 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 -> Peer d985fb669d8849c8af87bea35fd6dce5 (127.0.98.60:43969): Couldn't send request to peer d985fb669d8849c8af87bea35fd6dce5. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 20:49:54.313714 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c93f80caa404069886909d51fc4df50 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.313968 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c93f80caa404069886909d51fc4df50 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.314599 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.314672 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:54.314720 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.314756 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082 [sys.catalog]: This master's current role is: LEADER
I20260619 20:49:54.315013 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26d99f894e3242b48d8e02d1ee115082" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.315078 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:49:54.319988 7143 master.cc:552] Master@127.0.98.60:43969 shutdown complete.
I20260619 20:49:54.320036 7143 master_runner.cc:305] Clearing existing system tablet
I20260619 20:49:54.320164 7143 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:54.320209 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:54.320935 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:54.321517 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.321663 7143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:54.321722 7143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:49:54.321746 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
uuid: "d985fb669d8849c8af87bea35fd6dce5"
format_stamp: "Formatted at 2026-06-19 20:49:54 on dist-test-slave-0smx"
server_key: "c8295597b7d404b26995c1feaa76b648"
server_key_iv: "9af11c9cf64402377727c8a224ee4ffd"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.322420 7143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:54.323675 7143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 20:49:54.323740 7143 log.cc:1196] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 20:49:54.323925 7143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Deleting consensus metadata
I20260619 20:49:54.324041 7143 master_runner.cc:315] Copying system tablet from 127.0.98.62:38413
I20260619 20:49:54.324606 7143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38413
I20260619 20:49:54.336501 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.329653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.0.98.62:38413)
0619 20:49:54.329696 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.329699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.329811 (+ 112us) server_negotiation.cc:407] Connection header received
0619 20:49:54.329865 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.329866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.329876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.329944 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.330284 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.331041 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.332066 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.332356 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.332427 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.333533 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.333534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.333534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.333542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.335253 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.335648 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.335649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.335650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.335719 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.335936 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.335937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.335938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.336088 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.336165 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.336331 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.336406 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:49:54.336557 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.329590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:38230)
0619 20:49:54.329730 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.329742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.329808 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.329948 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.329950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.330152 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.330203 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.330206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.331138 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.331139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.331975 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.331978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.332444 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.332445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.332506 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.333021 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.333026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.333448 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.335342 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.335343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.335344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.335565 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.335798 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.335799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.335799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.335862 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.336169 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.336170 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.336216 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.336293 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.336451 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260619 20:49:54.337152 6512 tablet_copy_service.cc:140] P 26d99f894e3242b48d8e02d1ee115082: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d985fb669d8849c8af87bea35fd6dce5 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38230)
I20260619 20:49:54.337208 6512 tablet_copy_service.cc:161] P 26d99f894e3242b48d8e02d1ee115082: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d985fb669d8849c8af87bea35fd6dce5 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38230: session id = d985fb669d8849c8af87bea35fd6dce5-00000000000000000000000000000000
I20260619 20:49:54.337795 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 20:49:54.338231 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 20:49:54.339520 7143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: tablet copy: Starting download of 0 data blocks...
I20260619 20:49:54.339653 7143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: tablet copy: Starting download of 1 WAL segments...
I20260619 20:49:54.340420 7143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 20:49:54.341248 6512 tablet_copy_service.cc:342] P 26d99f894e3242b48d8e02d1ee115082: Request end of tablet copy session d985fb669d8849c8af87bea35fd6dce5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38230
I20260619 20:49:54.341413 6512 tablet_copy_service.cc:433] P 26d99f894e3242b48d8e02d1ee115082: ending tablet copy session d985fb669d8849c8af87bea35fd6dce5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d985fb669d8849c8af87bea35fd6dce5
I20260619 20:49:54.343997 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:54.345074 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.345121 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.345446 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:54.345705 7143 server_base.cc:1034] running on GCE node
I20260619 20:49:54.345824 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:54.345960 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:54.347110 7143 hybrid_clock.cc:648] HybridClock initialized: now 1781902194347102 us; error 26 us; skew 500 ppm
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:54.348555 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 20:49:54.348807 7143 webserver.cc:460] Webserver started at http://127.0.98.60:46557/ using document root <none> and password file <none>
I20260619 20:49:54.348925 7143 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:54.348958 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:54.349640 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:54.350216 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.350384 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:54.350435 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
uuid: "d985fb669d8849c8af87bea35fd6dce5"
format_stamp: "Formatted at 2026-06-19 20:49:54 on dist-test-slave-0smx"
server_key: "c8295597b7d404b26995c1feaa76b648"
server_key_iv: "9af11c9cf64402377727c8a224ee4ffd"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.350484 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:54.365545 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:54.367254 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:54.370134 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43969
I20260619 20:49:54.370218 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43969 every 8 connection(s)
I20260619 20:49:54.370445 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:49:54.374156 7312 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:49:54.374696 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Bootstrap starting.
I20260619 20:49:54.377765 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:49:54.377983 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: Bootstrap complete.
I20260619 20:49:54.378185 7312 raft_consensus.cc:348] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:49:54.378273 7312 raft_consensus.cc:729] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d985fb669d8849c8af87bea35fd6dce5, State: Initialized, Role: LEARNER
I20260619 20:49:54.378312 7312 consensus_queue.cc:260] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:49:54.378476 7312 sys_catalog.cc:564] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:49:54.378492 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } } }
I20260619 20:49:54.378685 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:49:54.380033 7326 catalog_manager.cc:1259] Loaded cluster ID: 8f8c6cd2327344528a6100074b26c604
I20260619 20:49:54.380088 7326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: loading cluster ID for follower catalog manager: success
I20260619 20:49:54.381735 7326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: acquiring CA information for follower catalog manager: success
I20260619 20:49:54.382020 7326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d985fb669d8849c8af87bea35fd6dce5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 20:49:54.392443 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:54.392693 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:54.392730 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:54.394042 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:54.394110 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:54.394165 7236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:49:54.395594 7236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41521
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:54.395850 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:54.396119 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:54.401022 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.401042 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.401043 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:54.401249 7236 server_base.cc:1034] running on GCE node
I20260619 20:49:54.401541 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:54.401804 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:54.402957 7236 hybrid_clock.cc:648] HybridClock initialized: now 1781902194402942 us; error 33 us; skew 500 ppm
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:54.405599 7236 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:49:54.406055 7236 webserver.cc:460] Webserver started at http://127.0.98.1:41521/ using document root <none> and password file <none>
I20260619 20:49:54.406322 7236 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:54.406404 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:54.407716 7236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:49:54.408296 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.408510 7236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:49:54.408605 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "191891f256bf49aa9ade310303d879b7"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "3f4d526ca0fa455039dd88cabbb41d23"
server_key_iv: "e738af2dfb3e670f46dcbdaaf9556b75"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.408999 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:54.428411 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:54.430214 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:54.430373 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:54.430588 7236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:54.430878 7236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:54.430933 7236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.430975 7236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:54.431003 7236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.437634 7236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42179
I20260619 20:49:54.437703 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42179 every 8 connection(s)
I20260619 20:49:54.438033 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:49:54.447063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 7236
I20260619 20:49:54.447227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6688
I20260619 20:49:54.448467 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.439051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46203 (local address 127.0.98.61:42575)
0619 20:49:54.439100 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.439102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.439113 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:49:54.439690 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.439691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.439704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.439735 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.440453 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.441171 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442247 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.442537 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442601 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.445123 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.445125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.445125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.445131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.446826 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.447323 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.447325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.447325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.447384 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.447660 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.447661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.447661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.447847 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.447890 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.448245 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.448324 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:49:54.449033 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.438948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.98.1:46203)
0619 20:49:54.439050 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.439059 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.439577 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.440191 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.440191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.440329 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.440382 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.440383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.441263 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.441264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.442153 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.442155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.442612 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.442705 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.443432 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.443437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.445040 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.446904 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.446909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.446912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.447227 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.447460 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.447461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.447461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.447584 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.447937 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.447944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.448101 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.448574 (+ 473us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.448752 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":70,"thread_start_us":65,"threads_started":1}
I20260619 20:49:54.451373 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42575
I20260619 20:49:54.451472 7457 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.451654 7457 heartbeater.cc:511] Master 127.0.98.61:42575 requested a full tablet report, sending...
I20260619 20:49:54.452186 7089 ts_manager.cc:194] Registered new tserver with Master: 191891f256bf49aa9ade310303d879b7 (127.0.98.1:42179)
I20260619 20:49:54.453366 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.438699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37167 (local address 127.0.98.60:43969)
0619 20:49:54.438848 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.438851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.439131 (+ 280us) server_negotiation.cc:407] Connection header received
0619 20:49:54.439724 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.439725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.439737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.439796 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.440409 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.441079 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442782 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.443040 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.443105 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.445618 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.445619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.445620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.445978 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.451947 (+ 5969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.452585 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.452586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.452587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.452639 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.452835 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.452836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.452837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.452967 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.453027 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.453179 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.453252 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260619 20:49:54.453374 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.438705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.98.1:37167)
0619 20:49:54.438869 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.438920 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.439572 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.439848 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.439851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.440154 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.440300 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.440306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.441184 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.441184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.442317 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.442320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.443111 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.443112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.443191 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.443590 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.443594 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.445531 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.452338 (+ 6807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.452341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.452341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.452496 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.452711 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.452711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.452712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.452768 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.453018 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.453019 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.453060 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.453142 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.453240 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":80,"thread_start_us":71,"threads_started":1}
I20260619 20:49:54.453601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37323
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38413
--webserver_port=35275
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:49:54.454211 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43969
I20260619 20:49:54.454262 7455 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.454332 7455 heartbeater.cc:511] Master 127.0.98.60:43969 requested a full tablet report, sending...
I20260619 20:49:54.454576 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.439266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49119 (local address 127.0.98.62:38413)
0619 20:49:54.439303 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.439304 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.439312 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:54.439682 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.439683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.439694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.439754 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.440402 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.441171 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442403 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.442689 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442740 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.446110 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.446111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.446112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.446119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.447647 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.453718 (+ 6071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.453719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.453719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.453775 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.453988 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.453989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.453990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.454124 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.454209 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.454412 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.454497 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:54.454655 7270 ts_manager.cc:194] Registered new tserver with Master: 191891f256bf49aa9ade310303d879b7 (127.0.98.1:42179)
I20260619 20:49:54.454857 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.439094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:49119)
0619 20:49:54.439179 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.439187 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.439575 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.439847 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.439851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.440114 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.440299 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.440303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.441235 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.441236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.442320 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.442321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.442757 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.442757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.442808 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.443348 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.443357 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.446016 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.453372 (+ 7356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.453373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.453374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.453621 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.453863 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.453863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.453864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.453922 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.454202 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.454204 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.454254 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.454350 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.454465 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":104,"thread_start_us":56,"threads_started":1}
I20260619 20:49:54.455261 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38413
I20260619 20:49:54.455302 7456 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.455376 7456 heartbeater.cc:511] Master 127.0.98.62:38413 requested a full tablet report, sending...
I20260619 20:49:54.455644 6492 ts_manager.cc:194] Re-registered known tserver with Master: 191891f256bf49aa9ade310303d879b7 (127.0.98.1:42179)
I20260619 20:49:54.456143 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49119
W20260619 20:49:54.536513 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:54.536701 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:54.536723 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:54.537942 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:54.537984 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:54.538034 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:49:54.539498 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35275
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:54.539732 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:54.539951 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:54.544699 7468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.544706 7469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:54.544740 7463 server_base.cc:1034] running on GCE node
W20260619 20:49:54.544744 7471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:54.545047 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:54.545326 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:54.546487 7463 hybrid_clock.cc:648] HybridClock initialized: now 1781902194546483 us; error 36 us; skew 500 ppm
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:54.548820 7463 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:49:54.549278 7463 webserver.cc:460] Webserver started at http://127.0.98.2:35275/ using document root <none> and password file <none>
I20260619 20:49:54.549539 7463 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:54.549623 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:54.550868 7463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:54.551411 7478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.551610 7463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:49:54.551718 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "1aea3357cc03b23441603b2e67d41d84"
server_key_iv: "5eb983499bcdd67126feafd6481c43b5"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.552038 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:54.572337 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:54.574365 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:54.574518 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:54.574740 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:54.575042 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:54.575096 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.575138 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:54.575165 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.581183 7463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37323
I20260619 20:49:54.581223 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37323 every 8 connection(s)
I20260619 20:49:54.581794 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:49:54.589221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 7463
I20260619 20:49:54.589402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6822
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 20:49:54.592985 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.582876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34409 (local address 127.0.98.61:42575)
0619 20:49:54.582931 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.582933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.582988 (+ 55us) server_negotiation.cc:407] Connection header received
0619 20:49:54.584157 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.584158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.584167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.584233 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.584909 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.585687 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.586746 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.587047 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.587110 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.589573 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.589574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.589574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.589581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.591326 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.591834 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.591835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.591836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.591898 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.592191 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.592192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.592193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.592348 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.592401 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.592785 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.592861 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:49:54.593840 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.582823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.98.2:34409)
0619 20:49:54.582925 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.582935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.583878 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.584273 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.584274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.584584 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.584805 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.584810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.585767 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.585767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.586665 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.586667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.587153 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.587154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.587200 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.587931 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.587935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.589447 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.591409 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.591416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.591418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.591739 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.591973 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.591975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.591975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.592110 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.592451 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.592458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.592624 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.593359 (+ 735us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.593603 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":42,"thread_start_us":61,"threads_started":1}
I20260619 20:49:54.594092 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.582548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59333 (local address 127.0.98.60:43969)
0619 20:49:54.582587 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.582589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.582820 (+ 231us) server_negotiation.cc:407] Connection header received
0619 20:49:54.584032 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.584033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.584047 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.584109 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.584919 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.585480 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.586774 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.587018 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.587080 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.591040 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.591042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.591042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.591049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.592775 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593172 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.593173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.593174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.593222 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593451 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.593452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.593452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.593622 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.593665 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.593854 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.593957 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:49:54.594411 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.582553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.98.2:59333)
0619 20:49:54.582692 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.582724 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.583870 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.584165 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.584168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.584589 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.584805 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.584811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.585577 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.585577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.586665 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.586667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.587129 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.587130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.587181 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.587848 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.587858 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.590942 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.592862 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.592863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.592863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.593100 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.593285 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.593286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.593384 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.593709 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.593710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.593751 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.594167 (+ 416us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.594271 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260619 20:49:54.594503 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42575
I20260619 20:49:54.594575 7594 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.594763 7594 heartbeater.cc:511] Master 127.0.98.61:42575 requested a full tablet report, sending...
I20260619 20:49:54.594957 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43969
I20260619 20:49:54.595005 7592 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.595047 7592 heartbeater.cc:511] Master 127.0.98.60:43969 requested a full tablet report, sending...
I20260619 20:49:54.595072 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.583042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57057 (local address 127.0.98.62:38413)
0619 20:49:54.583096 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.583099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.583139 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:49:54.584299 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.584301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.584312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.584388 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.584937 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.585718 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.586642 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.586958 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.587032 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.590953 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.590954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.590955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.590963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.593188 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593723 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.593724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.593724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.593789 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593996 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.593997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.593997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.594112 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.594156 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.594890 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.594968 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:49:54.595445 7270 ts_manager.cc:194] Registered new tserver with Master: 4789f58f8b4d4ab5a7fdebeb8c70b5a8 (127.0.98.2:37323)
I20260619 20:49:54.595445 7089 ts_manager.cc:194] Registered new tserver with Master: 4789f58f8b4d4ab5a7fdebeb8c70b5a8 (127.0.98.2:37323)
I20260619 20:49:54.596027 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.582981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:57057)
0619 20:49:54.583084 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.583094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.583875 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.584415 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.584416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.584635 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.584806 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.584811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.585803 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.585804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.586529 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.586531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.587050 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.587051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.587180 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.587868 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.587872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.590847 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.593272 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.593274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.593636 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.593860 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.593861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.593861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.593922 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.594711 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.594712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.594781 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.595652 (+ 871us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.595767 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":25,"thread_start_us":60,"threads_started":1}
I20260619 20:49:54.596483 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38413
I20260619 20:49:54.596539 7593 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.596599 7593 heartbeater.cc:511] Master 127.0.98.62:38413 requested a full tablet report, sending...
I20260619 20:49:54.596829 6492 ts_manager.cc:194] Re-registered known tserver with Master: 4789f58f8b4d4ab5a7fdebeb8c70b5a8 (127.0.98.2:37323)
I20260619 20:49:54.596837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34371
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38413
--webserver_port=43897
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:49:54.597399 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57057
W20260619 20:49:54.678642 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:49:54.678828 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:49:54.678848 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:49:54.680069 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:49:54.680110 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:49:54.680137 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:49:54.681633 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43897
--tserver_master_addrs=127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:49:54.681864 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:49:54.682099 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:49:54.686822 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.686879 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:49:54.686851 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:49:54.687188 7599 server_base.cc:1034] running on GCE node
I20260619 20:49:54.687354 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:49:54.687577 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:49:54.688735 7599 hybrid_clock.cc:648] HybridClock initialized: now 1781902194688728 us; error 35 us; skew 500 ppm
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:49:54.691146 7599 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:49:54.691536 7599 webserver.cc:460] Webserver started at http://127.0.98.3:43897/ using document root <none> and password file <none>
I20260619 20:49:54.691751 7599 fs_manager.cc:359] Metadata directory not provided
I20260619 20:49:54.691829 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:49:54.693125 7599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:49:54.693713 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.693871 7599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.693964 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "9c95658cbb6a4fdda85cfdac402e00b1"
format_stamp: "Formatted at 2026-06-19 20:49:53 on dist-test-slave-0smx"
server_key: "cc78196167d1cb3a9c03df66325b0be9"
server_key_iv: "8f4e8c92f1b65fe9c8ab1a80006c2725"
server_key_version: "encryptionkey@0"
I20260619 20:49:54.694314 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:49:54.705719 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:49:54.707387 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:49:54.707540 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:49:54.707751 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:49:54.708024 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:49:54.708076 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.708107 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:49:54.708132 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:49:54.714075 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34371
I20260619 20:49:54.714119 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34371 every 8 connection(s)
I20260619 20:49:54.714425 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 20:49:54 dist-test-slave-0smx krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 20:49:54.721942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 7599
I20260619 20:49:54.724843 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.715617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42705 (local address 127.0.98.62:38413)
0619 20:49:54.715662 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.715664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.715675 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:49:54.715961 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.715963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.715974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.716007 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.716743 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.717490 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718318 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.718577 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718650 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.720940 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.720942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.720942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.720950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.722763 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.723777 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.723778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.723778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.723837 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.724139 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.724140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.724140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.724279 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.724321 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.724687 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.724760 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:49:54.724865 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.715380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46049 (local address 127.0.98.60:43969)
0619 20:49:54.715424 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.715426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.715488 (+ 62us) server_negotiation.cc:407] Connection header received
0619 20:49:54.715961 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.715963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.715975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.716007 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.716669 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.717180 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718500 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.718780 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718843 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.721414 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.721416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.721416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.721425 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.723098 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.723859 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.723860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.723861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.723925 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.724123 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.724124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.724124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.724264 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.724315 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.724679 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.724756 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:49:54.725418 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.715309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43969 (local address 127.0.98.3:46049)
0619 20:49:54.715426 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.715436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.715864 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.716101 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.716102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.716402 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.716558 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.716565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.717275 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.717275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.718390 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.718392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.718891 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.718954 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.719477 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.719481 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.721344 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.723281 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.723281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.723282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.723503 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.723994 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.723995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.723995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.724056 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.724362 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.724369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.724517 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.724642 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.724800 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":6,"thread_start_us":58,"threads_started":1}
I20260619 20:49:54.726089 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43969
I20260619 20:49:54.726173 7728 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.726192 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.715083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44963 (local address 127.0.98.61:42575)
0619 20:49:54.715147 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.715149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.715323 (+ 174us) server_negotiation.cc:407] Connection header received
0619 20:49:54.715946 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.715947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.715959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.716007 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.716655 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.717463 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718472 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.718752 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718822 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.723314 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.723315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.723315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.723323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.724868 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.725356 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.725357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.725358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.725418 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.725633 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.725634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.725634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.725762 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.725803 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.726031 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.726104 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:49:54.726344 7728 heartbeater.cc:511] Master 127.0.98.60:43969 requested a full tablet report, sending...
I20260619 20:49:54.726491 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.715038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42575 (local address 127.0.98.3:44963)
0619 20:49:54.715184 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.715222 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.715839 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.716069 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.716073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.716383 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.716557 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.716561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.717540 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.717540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.718402 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.718404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.718823 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.718898 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.719542 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.719547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.723224 (+ 3677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.724996 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.724997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.724997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.725232 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.725502 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.725503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.725503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.725562 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.725851 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.725851 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.725902 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.726287 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.726408 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":57,"thread_start_us":86,"threads_started":1}
I20260619 20:49:54.727090 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.715453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:42705)
0619 20:49:54.715547 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.715555 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.715863 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.716468 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.716469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.716609 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:49:54.716665 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.716666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.717573 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.717574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.718232 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.718234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.718661 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.718662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.718794 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.719250 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:49:54.719259 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:49:54.720851 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:49:54.722837 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.722843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.722845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.723203 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.723906 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:49:54.723907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:49:54.723907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:49:54.724052 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:49:54.724371 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:49:54.724372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:49:54.724517 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.724961 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.726348 (+ 1387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":27,"thread_start_us":58,"threads_started":1}
I20260619 20:49:54.727239 7270 ts_manager.cc:194] Registered new tserver with Master: 9c95658cbb6a4fdda85cfdac402e00b1 (127.0.98.3:34371)
I20260619 20:49:54.727644 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42575
I20260619 20:49:54.727690 7730 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.727744 7730 heartbeater.cc:511] Master 127.0.98.61:42575 requested a full tablet report, sending...
I20260619 20:49:54.728005 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38413
I20260619 20:49:54.728068 7729 heartbeater.cc:464] Registering TS with master...
I20260619 20:49:54.728133 7729 heartbeater.cc:511] Master 127.0.98.62:38413 requested a full tablet report, sending...
I20260619 20:49:54.728381 7089 ts_manager.cc:194] Registered new tserver with Master: 9c95658cbb6a4fdda85cfdac402e00b1 (127.0.98.3:34371)
I20260619 20:49:54.728405 6492 ts_manager.cc:194] Re-registered known tserver with Master: 9c95658cbb6a4fdda85cfdac402e00b1 (127.0.98.3:34371)
I20260619 20:49:54.728866 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42705
I20260619 20:49:54.733057 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.723654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.0.98.62:38413)
0619 20:49:54.724130 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.724132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.724140 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:49:54.724160 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.724162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.724173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.725759 (+ 1586us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.725795 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.726534 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.727988 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.728265 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.728318 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.729893 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.729894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.729895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.729901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.731811 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.732184 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.732185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.732186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.732250 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.732506 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.732508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.732508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.732641 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.732765 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.732865 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.732941 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260619 20:49:54.734908 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.723559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.0.98.61:42575)
0619 20:49:54.725091 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.725093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.725100 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:54.725117 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.725118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.725128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.725206 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.727977 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.728633 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.729795 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.730128 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.730172 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.731776 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.731778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.731778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.731784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.733724 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.734141 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.734142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.734143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.734203 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.734424 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.734425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.734425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.734555 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.734662 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.734752 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.734826 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1515,"thread_start_us":54,"threads_started":1}
I20260619 20:49:54.735400 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.723563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.0.98.60:43969)
0619 20:49:54.723809 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.723810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.725366 (+ 1556us) server_negotiation.cc:407] Connection header received
0619 20:49:54.725457 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.725458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.725470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.725527 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.726305 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.727019 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.729828 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.730074 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.730153 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.731597 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.731598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.731599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.731605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.733338 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.733829 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.733835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.733835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.733879 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.734110 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.734112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.734112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.734441 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.734533 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.735232 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.735320 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":61,"threads_started":1}
I20260619 20:49:54.734759 6492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:49:54.736685 6492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:49:54.743947 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.740170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34371 (local address 127.0.0.1:40394)
0619 20:49:54.740442 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.740461 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.740487 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.740769 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.740770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.740771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:54.740817 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.740819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.741796 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.741797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.742784 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.742786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.743690 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.743691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.743750 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.743754 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.743823 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.743868 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":87,"threads_started":1}
I20260619 20:49:54.743957 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.740243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.0.98.3:34371)
0619 20:49:54.740585 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.740589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.740598 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:54.740630 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.740632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.740647 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.740885 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:54.740933 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.741702 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.742850 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.743600 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.743679 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.743748 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.743836 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.743880 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":70,"threads_started":1}
I20260619 20:49:54.744185 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.740109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42179 (local address 127.0.0.1:42398)
0619 20:49:54.740478 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.740488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.740539 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.740734 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.740735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.740737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:54.740785 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.740787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.742083 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.742084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.743137 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.743140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.743958 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.743959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.744035 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.744039 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.744085 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.744119 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":226,"thread_start_us":84,"threads_started":1}
I20260619 20:49:54.744391 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.740218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.0.98.1:42179)
0619 20:49:54.740537 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.740543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.740554 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:49:54.740597 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.740600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.740615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.740736 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:54.740865 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.741995 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.743222 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.743791 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.743859 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.743928 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.744273 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.744327 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":74,"threads_started":1}
I20260619 20:49:54.744725 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.740109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37323 (local address 127.0.0.1:60200)
0619 20:49:54.740296 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.740306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.740345 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.741020 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.741021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.741022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:54.741057 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.741058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.742301 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.742303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.743200 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.743202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.744586 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.744587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.744628 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.744630 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.744665 (+ 35us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.744689 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 20:49:54.744804 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.740251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.0.98.2:37323)
0619 20:49:54.740847 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.740852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.740861 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:54.740909 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.740911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.740925 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.741122 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:54.741201 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.742223 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.743305 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.744401 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.744591 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.744663 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.744689 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.744731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":71,"threads_started":1}
I20260619 20:49:54.744685 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 56ce3095130d4bc7b34fe67f0cea1f02 (DEFAULT_TABLE table=test-table [id=089bcf8cf1bb4e3cae0a01f34fb68bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:54.745057 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ce3095130d4bc7b34fe67f0cea1f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:54.745426 7526 tablet_service.cc:1467] Processing CreateTablet for tablet 56ce3095130d4bc7b34fe67f0cea1f02 (DEFAULT_TABLE table=test-table [id=089bcf8cf1bb4e3cae0a01f34fb68bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:54.745803 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ce3095130d4bc7b34fe67f0cea1f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:54.746017 7389 tablet_service.cc:1467] Processing CreateTablet for tablet 56ce3095130d4bc7b34fe67f0cea1f02 (DEFAULT_TABLE table=test-table [id=089bcf8cf1bb4e3cae0a01f34fb68bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:49:54.746340 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ce3095130d4bc7b34fe67f0cea1f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:49:54.748462 7754 tablet_bootstrap.cc:492] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Bootstrap starting.
I20260619 20:49:54.748925 7755 tablet_bootstrap.cc:492] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Bootstrap starting.
I20260619 20:49:54.749190 7754 tablet_bootstrap.cc:654] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:54.749632 7755 tablet_bootstrap.cc:654] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:54.750070 7755 log.cc:824] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:54.750109 7754 log.cc:824] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:54.750371 7756 tablet_bootstrap.cc:492] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Bootstrap starting.
I20260619 20:49:54.750785 7755 tablet_bootstrap.cc:492] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: No bootstrap required, opened a new log
I20260619 20:49:54.750850 7755 ts_tablet_manager.cc:1397] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:49:54.750818 7754 tablet_bootstrap.cc:492] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: No bootstrap required, opened a new log
I20260619 20:49:54.750929 7754 ts_tablet_manager.cc:1397] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 20:49:54.751060 7756 tablet_bootstrap.cc:654] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Neither blocks nor log segments found. Creating new log.
I20260619 20:49:54.751346 7756 log.cc:824] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:49:54.752102 7756 tablet_bootstrap.cc:492] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: No bootstrap required, opened a new log
I20260619 20:49:54.752161 7756 ts_tablet_manager.cc:1397] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:49:54.752890 7754 raft_consensus.cc:348] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.752959 7755 raft_consensus.cc:348] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.753034 7754 raft_consensus.cc:374] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:54.753063 7755 raft_consensus.cc:374] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:54.753067 7754 raft_consensus.cc:729] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4789f58f8b4d4ab5a7fdebeb8c70b5a8, State: Initialized, Role: FOLLOWER
I20260619 20:49:54.753083 7755 raft_consensus.cc:729] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c95658cbb6a4fdda85cfdac402e00b1, State: Initialized, Role: FOLLOWER
I20260619 20:49:54.753196 7755 consensus_queue.cc:260] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.753213 7754 consensus_queue.cc:260] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.753497 7729 heartbeater.cc:503] Master 127.0.98.62:38413 was elected leader, sending a full tablet report...
I20260619 20:49:54.753481 7755 ts_tablet_manager.cc:1428] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:49:54.754426 7754 ts_tablet_manager.cc:1428] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 20:49:54.754541 7593 heartbeater.cc:503] Master 127.0.98.62:38413 was elected leader, sending a full tablet report...
I20260619 20:49:54.754619 7756 raft_consensus.cc:348] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.754748 7756 raft_consensus.cc:374] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:49:54.754786 7756 raft_consensus.cc:729] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191891f256bf49aa9ade310303d879b7, State: Initialized, Role: FOLLOWER
I20260619 20:49:54.755312 7756 consensus_queue.cc:260] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.755580 7756 ts_tablet_manager.cc:1428] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:49:54.755774 7456 heartbeater.cc:503] Master 127.0.98.62:38413 was elected leader, sending a full tablet report...
I20260619 20:49:54.829403 7762 raft_consensus.cc:482] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:49:54.829516 7762 raft_consensus.cc:504] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.829931 7762 leader_election.cc:288] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4789f58f8b4d4ab5a7fdebeb8c70b5a8 (127.0.98.2:37323), 9c95658cbb6a4fdda85cfdac402e00b1 (127.0.98.3:34371)
W20260619 20:49:54.832576 7595 tablet.cc:2369] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:49:54.832875 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.830122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34371 (local address 127.0.98.1:51421)
0619 20:49:54.830176 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.830190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.830312 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.830517 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.830518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.830519 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:54.830579 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.830580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.831215 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.831215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.832009 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.832011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.832637 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.832638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.832708 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.832711 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.832758 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.832795 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":3}
I20260619 20:49:54.832865 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.830311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51421 (local address 127.0.98.3:34371)
0619 20:49:54.830371 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.830373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.830380 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:49:54.830424 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.830425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.830434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.830502 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:54.830671 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.831134 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.832100 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.832548 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.832636 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.832647 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.832770 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.832804 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:49:54.833206 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ce3095130d4bc7b34fe67f0cea1f02" candidate_uuid: "191891f256bf49aa9ade310303d879b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" is_pre_election: true
I20260619 20:49:54.833433 7682 raft_consensus.cc:2454] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191891f256bf49aa9ade310303d879b7 in term 0.
I20260619 20:49:54.833508 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.830122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37323 (local address 127.0.98.1:46661)
0619 20:49:54.830245 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:54.830273 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 20:49:54.830318 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:49:54.830480 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:49:54.830481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:49:54.830483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:49:54.830564 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.830566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.831214 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.831215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.832007 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:49:54.832009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.833214 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.833215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:49:54.833273 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.833275 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:49:54.833398 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:49:54.833441 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":92}
I20260619 20:49:54.833518 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.830235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46661 (local address 127.0.98.2:37323)
0619 20:49:54.830302 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.830304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.830316 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:49:54.830385 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.830386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.830412 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.830466 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:49:54.830628 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.831135 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.832098 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.833140 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.833210 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.833221 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.833425 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.833464 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:49:54.833675 7345 leader_election.cc:302] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191891f256bf49aa9ade310303d879b7, 9c95658cbb6a4fdda85cfdac402e00b1; no voters:
I20260619 20:49:54.833824 7762 raft_consensus.cc:2781] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:49:54.833878 7762 raft_consensus.cc:482] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:49:54.833901 7762 raft_consensus.cc:3037] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:54.833843 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ce3095130d4bc7b34fe67f0cea1f02" candidate_uuid: "191891f256bf49aa9ade310303d879b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" is_pre_election: true
I20260619 20:49:54.833973 7546 raft_consensus.cc:2454] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191891f256bf49aa9ade310303d879b7 in term 0.
I20260619 20:49:54.834725 7762 raft_consensus.cc:504] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.834854 7762 leader_election.cc:288] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4789f58f8b4d4ab5a7fdebeb8c70b5a8 (127.0.98.2:37323), 9c95658cbb6a4fdda85cfdac402e00b1 (127.0.98.3:34371)
I20260619 20:49:54.835012 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ce3095130d4bc7b34fe67f0cea1f02" candidate_uuid: "191891f256bf49aa9ade310303d879b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c95658cbb6a4fdda85cfdac402e00b1"
I20260619 20:49:54.835047 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ce3095130d4bc7b34fe67f0cea1f02" candidate_uuid: "191891f256bf49aa9ade310303d879b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8"
I20260619 20:49:54.835125 7546 raft_consensus.cc:3037] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:54.835125 7682 raft_consensus.cc:3037] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:49:54.836025 7546 raft_consensus.cc:2454] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191891f256bf49aa9ade310303d879b7 in term 1.
I20260619 20:49:54.836025 7682 raft_consensus.cc:2454] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191891f256bf49aa9ade310303d879b7 in term 1.
I20260619 20:49:54.836189 7344 leader_election.cc:302] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191891f256bf49aa9ade310303d879b7, 4789f58f8b4d4ab5a7fdebeb8c70b5a8; no voters:
I20260619 20:49:54.836270 7762 raft_consensus.cc:2781] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:49:54.836416 7762 raft_consensus.cc:686] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 1 LEADER]: Becoming Leader. State: Replica: 191891f256bf49aa9ade310303d879b7, State: Running, Role: LEADER
I20260619 20:49:54.836613 7762 consensus_queue.cc:237] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } }
I20260619 20:49:54.837467 6492 catalog_manager.cc:5495] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191891f256bf49aa9ade310303d879b7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "191891f256bf49aa9ade310303d879b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191891f256bf49aa9ade310303d879b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 } health_report { overall_health: UNKNOWN } } }
I20260619 20:49:54.862057 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.859508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.0.98.1:42179)
0619 20:49:54.859566 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.859569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.859578 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:49:54.859657 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.859657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.859675 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.859764 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:49:54.859900 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.860346 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.861223 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.861474 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.861557 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.861730 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:49:54.861832 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:49:54.861918 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.861956 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.861975 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:49:54.864501 7546 raft_consensus.cc:1261] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 1 FOLLOWER]: Refusing update from remote peer 191891f256bf49aa9ade310303d879b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:54.864503 7682 raft_consensus.cc:1261] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 1 FOLLOWER]: Refusing update from remote peer 191891f256bf49aa9ade310303d879b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:49:54.864845 7763 consensus_queue.cc:1035] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4789f58f8b4d4ab5a7fdebeb8c70b5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:54.864988 7762 consensus_queue.cc:1035] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c95658cbb6a4fdda85cfdac402e00b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:49:54.867609 7766 mvcc.cc:205] Tried to move back new op lower bound from 7298671390162321408 to 7298671390051446784. Current Snapshot: MvccSnapshot[applied={T|T < 7298671390162321408}]
I20260619 20:49:54.867761 7767 mvcc.cc:205] Tried to move back new op lower bound from 7298671390162321408 to 7298671390051446784. Current Snapshot: MvccSnapshot[applied={T|T < 7298671390162321408}]
I20260619 20:49:54.869661 7768 mvcc.cc:205] Tried to move back new op lower bound from 7298671390162321408 to 7298671390051446784. Current Snapshot: MvccSnapshot[applied={T|T < 7298671390162321408}]
I20260619 20:49:54.870002 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:54.870122 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:49:54.871541 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d99f894e3242b48d8e02d1ee115082: Sending DeleteTablet for 3 replicas of tablet 56ce3095130d4bc7b34fe67f0cea1f02
I20260619 20:49:54.871814 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ce3095130d4bc7b34fe67f0cea1f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42398
I20260619 20:49:54.871872 7526 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ce3095130d4bc7b34fe67f0cea1f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60200
I20260619 20:49:54.872078 7776 tablet_replica.cc:330] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: stopping tablet replica
I20260619 20:49:54.872265 7776 raft_consensus.cc:2229] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:49:54.872311 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ce3095130d4bc7b34fe67f0cea1f02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:49:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40394
I20260619 20:49:54.872501 7776 raft_consensus.cc:2258] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:54.872848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6460
I20260619 20:49:54.873565 7776 ts_tablet_manager.cc:1905] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:54.874595 7779 tablet_replica.cc:330] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: stopping tablet replica
I20260619 20:49:54.874719 7779 raft_consensus.cc:2229] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:54.874833 7779 raft_consensus.cc:2258] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:54.875679 7776 ts_tablet_manager.cc:1918] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:54.876951 7776 log.cc:1196] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/56ce3095130d4bc7b34fe67f0cea1f02
I20260619 20:49:54.877312 7779 ts_tablet_manager.cc:1905] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:49:54.877481 7776 ts_tablet_manager.cc:1939] T 56ce3095130d4bc7b34fe67f0cea1f02 P 191891f256bf49aa9ade310303d879b7: Deleting consensus metadata
I20260619 20:49:54.878667 7777 tablet_replica.cc:330] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: stopping tablet replica
I20260619 20:49:54.878800 7777 raft_consensus.cc:2229] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:49:54.878903 7777 raft_consensus.cc:2258] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:49:54.879774 7777 ts_tablet_manager.cc:1905] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:49:54.880263 7343 connection.cc:535] server connection from 127.0.0.1:42398 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:49:54.881337 7779 ts_tablet_manager.cc:1918] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:54.881420 7779 log.cc:1196] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/56ce3095130d4bc7b34fe67f0cea1f02
I20260619 20:49:54.881742 7779 ts_tablet_manager.cc:1939] T 56ce3095130d4bc7b34fe67f0cea1f02 P 9c95658cbb6a4fdda85cfdac402e00b1: Deleting consensus metadata
W20260619 20:49:54.882165 7617 connection.cc:441] server connection from 127.0.0.1:40394 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 1) could send its response
I20260619 20:49:54.883353 7777 ts_tablet_manager.cc:1918] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:49:54.883445 7777 log.cc:1196] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/56ce3095130d4bc7b34fe67f0cea1f02
I20260619 20:49:54.883760 7777 ts_tablet_manager.cc:1939] T 56ce3095130d4bc7b34fe67f0cea1f02 P 4789f58f8b4d4ab5a7fdebeb8c70b5a8: Deleting consensus metadata
W20260619 20:49:54.884091 7481 connection.cc:441] server connection from 127.0.0.1:60200 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60200 (request call id 1) could send its response
I20260619 20:49:54.886646 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.872460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.0.98.60:43969)
0619 20:49:54.872523 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.872525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.872768 (+ 243us) server_negotiation.cc:407] Connection header received
0619 20:49:54.872877 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.872878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.872893 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.872926 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.874995 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.875760 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.879176 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.879439 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.879480 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.882679 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.882681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.882681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.882689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.884946 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.885442 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.885443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.885443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.885503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.885709 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.885710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.885710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.885824 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.885912 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.886459 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.886546 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:49:54.890411 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.882103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.0.98.61:42575)
0619 20:49:54.882152 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.882154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.882794 (+ 640us) server_negotiation.cc:407] Connection header received
0619 20:49:54.882902 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.882903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.882915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.882945 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.883547 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.884307 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.885347 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.885642 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.885684 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.887128 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.887129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.887130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.887136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.889102 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.889561 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.889562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.889562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.889624 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.889836 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.889837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.889838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.889958 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.890084 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.890207 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.890291 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:49:54.891445 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:54.882467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.0.98.60:43969)
0619 20:49:54.882501 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:49:54.882503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:49:54.884326 (+ 1823us) server_negotiation.cc:407] Connection header received
0619 20:49:54.884434 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:49:54.884435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:49:54.884446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:49:54.884482 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:49:54.885115 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.885825 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.886892 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:49:54.887169 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:49:54.887220 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:49:54.888571 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:49:54.888573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:49:54.888573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:49:54.888580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:49:54.890339 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.890740 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.890741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.890741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.890797 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:49:54.891008 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:49:54.891009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:49:54.891010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:49:54.891124 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:49:54.891206 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:49:54.891328 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 20:49:54.891390 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:49:55.869711 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:55.869418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:38531)
0619 20:49:55.869568 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:55.869651 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260619 20:49:55.869710 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:55.869452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:59009)
0619 20:49:55.869559 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:55.869662 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":44,"threads_started":1}
W20260619 20:49:55.869958 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
W20260619 20:49:55.870113 7593 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:49:55.873066 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:55.872857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:44373)
0619 20:49:55.872963 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:55.873024 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
W20260619 20:49:55.873322 7729 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:49:56.372285 7790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:49:56.372424 7790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:49:56.372732 7790 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:49:56.373037 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:56.372878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:43346)
0619 20:49:56.372958 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:56.373005 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":47,"threads_started":1}
W20260619 20:49:56.373183 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:49:56.373620 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:56.373514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:43352)
0619 20:49:56.373571 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:56.373601 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:49:56.373706 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:49:56.373761 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:49:56.373844 7790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:49:56.870888 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:56.870678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:53571)
0619 20:49:56.870781 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:56.870823 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260619 20:49:56.871294 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:56.870302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:51691)
0619 20:49:56.870955 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:56.871245 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":53,"threads_started":1}
I20260619 20:49:56.873827 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:56.873641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:57845)
0619 20:49:56.873739 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:56.873776 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20260619 20:49:57.871944 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:57.871383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:38941)
0619 20:49:57.871580 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:57.871628 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":63,"threads_started":1}
I20260619 20:49:57.872102 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:57.871871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:59667)
0619 20:49:57.872014 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:57.872052 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":46,"threads_started":1}
I20260619 20:49:57.874310 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:57.874175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:37619)
0619 20:49:57.874256 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:57.874278 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":40,"threads_started":1}
I20260619 20:49:58.237144 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:49:58.237233 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:49:58.237401 7799 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:49:58.237756 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:58.237563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:43618)
0619 20:49:58.237676 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:58.237717 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260619 20:49:58.238153 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:58.238065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:43630)
0619 20:49:58.238109 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:58.238134 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:49:58.238231 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:49:58.238276 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:49:58.238343 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:49:58.872656 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:58.872423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:55491)
0619 20:49:58.872558 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:58.872597 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260619 20:49:58.872730 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:58.872559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:47749)
0619 20:49:58.872651 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:58.872686 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
I20260619 20:49:58.874794 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:58.874638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:57333)
0619 20:49:58.874731 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:58.874763 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
I20260619 20:49:59.873379 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:59.873169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:52837)
0619 20:49:59.873286 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:59.873341 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260619 20:49:59.873358 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:59.873108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:40821)
0619 20:49:59.873226 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:59.873268 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260619 20:49:59.875332 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:49:59.875137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:48707)
0619 20:49:59.875271 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:49:59.875298 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260619 20:50:00.874136 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:00.873914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:46515)
0619 20:50:00.874068 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:00.874099 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":41,"threads_started":1}
I20260619 20:50:00.874116 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:00.873870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:54117)
0619 20:50:00.874006 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:00.874052 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260619 20:50:00.875860 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:00.875689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:44095)
0619 20:50:00.875794 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:00.875828 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
I20260619 20:50:01.244710 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:50:01.244828 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:50:01.244967 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:50:01.245379 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:01.245152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:43946)
0619 20:50:01.245247 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:01.245325 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":53,"threads_started":1}
I20260619 20:50:01.245847 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:01.245751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:43960)
0619 20:50:01.245798 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:01.245823 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:50:01.245929 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:01.245967 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:50:01.246057 7811 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:50:01.874785 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:01.874523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:46633)
0619 20:50:01.874680 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:01.874729 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260619 20:50:01.874816 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:01.874523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:57487)
0619 20:50:01.874749 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:01.874780 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":67,"threads_started":1}
I20260619 20:50:01.876327 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:01.876171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:53579)
0619 20:50:01.876263 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:01.876294 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260619 20:50:02.875528 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:02.875317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:40383)
0619 20:50:02.875437 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:02.875477 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260619 20:50:02.875555 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:02.875309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:58935)
0619 20:50:02.875476 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:02.875513 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260619 20:50:02.876926 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:02.876727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:50125)
0619 20:50:02.876861 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:02.876892 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260619 20:50:03.876240 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:03.876019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:39427)
0619 20:50:03.876132 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:03.876171 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260619 20:50:03.876230 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:03.875986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:58817)
0619 20:50:03.876131 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:03.876173 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":104,"threads_started":1}
I20260619 20:50:03.877451 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:03.877258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:49761)
0619 20:50:03.877394 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:03.877418 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260619 20:50:04.876986 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:04.876714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:57417)
0619 20:50:04.876862 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:04.876914 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260619 20:50:04.877045 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:04.876750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:57789)
0619 20:50:04.876956 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:04.876988 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":58,"threads_started":1}
I20260619 20:50:04.877990 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:04.877812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:41127)
0619 20:50:04.877915 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:04.877948 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":69,"threads_started":1}
I20260619 20:50:05.161828 7826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:50:05.161934 7826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:50:05.162055 7826 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:50:05.162391 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:05.162220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:57114)
0619 20:50:05.162312 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:05.162354 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":49,"threads_started":1}
W20260619 20:50:05.162470 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260619 20:50:05.162820 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:05.162733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:57122)
0619 20:50:05.162777 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:05.162803 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:50:05.162894 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:05.162966 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:50:05.163048 7826 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:50:05.877714 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:05.877459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:55239)
0619 20:50:05.877561 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:05.877661 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260619 20:50:05.877714 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:05.877446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:47977)
0619 20:50:05.877608 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:05.877652 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260619 20:50:05.878487 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:05.878271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:33839)
0619 20:50:05.878419 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:05.878454 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":43,"threads_started":1}
I20260619 20:50:06.878397 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:06.878163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:57415)
0619 20:50:06.878296 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:06.878340 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260619 20:50:06.878427 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:06.878163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:50129)
0619 20:50:06.878304 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:06.878379 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260619 20:50:06.878981 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:06.878831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:46327)
0619 20:50:06.878919 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:06.878949 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
I20260619 20:50:07.879091 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:07.878867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:33889)
0619 20:50:07.878999 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:07.879037 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260619 20:50:07.879235 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:07.878956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:37117)
0619 20:50:07.879116 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:07.879151 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":51,"threads_started":1}
I20260619 20:50:07.879513 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:07.879321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:38597)
0619 20:50:07.879446 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:07.879481 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":51,"threads_started":1}
I20260619 20:50:08.879845 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:08.879624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:37665)
0619 20:50:08.879744 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:08.879786 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260619 20:50:08.879878 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:08.879648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:54135)
0619 20:50:08.879768 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:08.879825 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260619 20:50:08.880117 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:08.879881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:49247)
0619 20:50:08.880010 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:08.880077 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":52,"threads_started":1}
I20260619 20:50:09.880605 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:09.880361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:57917)
0619 20:50:09.880523 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:09.880556 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260619 20:50:09.880678 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:09.880354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:47681)
0619 20:50:09.880470 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:09.880508 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260619 20:50:09.880652 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:09.880484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:43605)
0619 20:50:09.880572 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:09.880605 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
I20260619 20:50:10.881345 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:10.881080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:45491)
0619 20:50:10.881236 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:10.881273 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":48,"threads_started":1}
I20260619 20:50:10.881356 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:10.881108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:56251)
0619 20:50:10.881225 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:10.881272 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260619 20:50:10.881388 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:10.881002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:52755)
0619 20:50:10.881140 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:10.881182 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260619 20:50:11.426613 7847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:50:11.426700 7847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:50:11.426890 7847 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:50:11.427227 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:11.427049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:57484)
0619 20:50:11.427147 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:11.427189 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":58,"threads_started":1}
W20260619 20:50:11.427304 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:11.427649 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:11.427558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:57492)
0619 20:50:11.427606 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:11.427632 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:50:11.427721 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:11.427798 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:50:11.427893 7847 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:50:11.881986 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:11.881744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:45857)
0619 20:50:11.881881 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:11.881926 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260619 20:50:11.882072 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:11.881744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:41211)
0619 20:50:11.881997 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:11.882030 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
I20260619 20:50:11.882220 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:11.881866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:44019)
0619 20:50:11.882134 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:11.882174 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":49,"threads_started":1}
I20260619 20:50:12.882611 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:12.882392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:41341)
0619 20:50:12.882511 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:12.882551 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260619 20:50:12.882691 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:12.882393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:40045)
0619 20:50:12.882621 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:12.882652 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":55,"threads_started":1}
I20260619 20:50:12.882815 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:12.882557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:42475)
0619 20:50:12.882732 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:12.882767 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":40,"threads_started":1}
I20260619 20:50:13.883356 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:13.883153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:36267)
0619 20:50:13.883294 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:13.883323 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":42,"threads_started":1}
I20260619 20:50:13.883350 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:13.883106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:59415)
0619 20:50:13.883262 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:13.883301 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260619 20:50:13.883350 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:13.883093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:41657)
0619 20:50:13.883260 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:13.883299 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":50,"threads_started":1}
I20260619 20:50:14.884002 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:14.883762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:44389)
0619 20:50:14.883921 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:14.883950 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260619 20:50:14.884002 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:14.883769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:52011)
0619 20:50:14.883906 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:14.883950 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260619 20:50:14.884009 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:14.883793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:45503)
0619 20:50:14.883912 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:14.883964 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 20:50:15.884658 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:15.884390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:40415)
0619 20:50:15.884535 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:15.884598 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260619 20:50:15.884685 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:15.884468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:44173)
0619 20:50:15.884627 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:15.884652 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":45,"threads_started":1}
I20260619 20:50:15.884684 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:15.884463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:49223)
0619 20:50:15.884603 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:15.884638 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":55,"threads_started":1}
I20260619 20:50:16.885337 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:16.885067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:40075)
0619 20:50:16.885236 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:16.885264 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":61,"threads_started":1}
I20260619 20:50:16.885339 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:16.885069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:43613)
0619 20:50:16.885218 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:16.885261 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260619 20:50:16.885378 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:16.885134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:50587)
0619 20:50:16.885251 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:16.885331 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":42,"threads_started":1}
I20260619 20:50:17.885928 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:17.885711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:49091)
0619 20:50:17.885830 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:17.885868 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260619 20:50:17.885924 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:17.885712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:50613)
0619 20:50:17.885856 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:17.885880 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260619 20:50:17.886217 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:17.885901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:50521)
0619 20:50:17.886111 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:17.886156 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260619 20:50:18.886621 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:18.886396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:37863)
0619 20:50:18.886518 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:18.886561 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260619 20:50:18.886684 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:18.886488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:60127)
0619 20:50:18.886612 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:18.886640 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20260619 20:50:18.886858 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:18.886540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:58887)
0619 20:50:18.886765 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:18.886804 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":46,"threads_started":1}
I20260619 20:50:19.471503 7874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:50:19.471601 7874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:50:19.471765 7874 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:50:19.472072 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:19.471900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:60166)
0619 20:50:19.471994 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:19.472033 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
W20260619 20:50:19.472213 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:19.472609 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:19.472501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:60174)
0619 20:50:19.472553 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:19.472582 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:50:19.472714 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:19.472800 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:50:19.472889 7874 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:50:19.887431 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:19.887062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:41531)
0619 20:50:19.887214 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:19.887257 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260619 20:50:19.887454 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:19.887241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:39133)
0619 20:50:19.887363 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:19.887400 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260619 20:50:19.887492 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:19.887295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:39415)
0619 20:50:19.887421 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:19.887452 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260619 20:50:20.888118 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:20.887857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:60471)
0619 20:50:20.887986 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:20.888063 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260619 20:50:20.888199 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:20.887982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:45761)
0619 20:50:20.888117 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:20.888154 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":41,"threads_started":1}
I20260619 20:50:20.888276 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:20.887846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:51869)
0619 20:50:20.888120 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:20.888169 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":68,"threads_started":1}
I20260619 20:50:21.809783 7883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26d99f894e3242b48d8e02d1ee115082)
I20260619 20:50:21.809885 7883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "26d99f894e3242b48d8e02d1ee115082" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38413 } } peers { permanent_uuid: "0c93f80caa404069886909d51fc4df50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42575 } attrs { promote: false } } peers { permanent_uuid: "d985fb669d8849c8af87bea35fd6dce5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43969 } attrs { promote: true } }
I20260619 20:50:21.810053 7883 leader_election.cc:288] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413)
I20260619 20:50:21.810429 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:21.810241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:60276)
0619 20:50:21.810352 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:21.810394 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":76,"threads_started":1}
I20260619 20:50:21.810842 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:21.810740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.0.1:60290)
0619 20:50:21.810799 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:21.810825 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 20:50:21.810916 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26d99f894e3242b48d8e02d1ee115082 (127.0.98.62:38413): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
I20260619 20:50:21.810968 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c93f80caa404069886909d51fc4df50; no voters: 26d99f894e3242b48d8e02d1ee115082
I20260619 20:50:21.811034 7883 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c93f80caa404069886909d51fc4df50 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:50:21.888819 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:21.888526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:56583)
0619 20:50:21.888709 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:21.888749 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260619 20:50:21.888841 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:21.888623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:40327)
0619 20:50:21.888740 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:21.888787 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
I20260619 20:50:21.888868 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:21.888549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:43795)
0619 20:50:21.888792 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:21.888825 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":76,"threads_started":1}
I20260619 20:50:22.889448 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:22.889192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:37371)
0619 20:50:22.889366 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:22.889396 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260619 20:50:22.889448 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:22.889191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:37767)
0619 20:50:22.889355 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:22.889396 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260619 20:50:22.889544 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:22.889260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:40539)
0619 20:50:22.889438 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:22.889491 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
I20260619 20:50:23.890084 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:23.889858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:51965)
0619 20:50:23.889990 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:23.890029 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260619 20:50:23.890163 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:23.889941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:42309)
0619 20:50:23.890082 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:23.890115 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260619 20:50:23.890199 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:23.889961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:49715)
0619 20:50:23.890133 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:23.890158 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":48,"threads_started":1}
I20260619 20:50:24.890722 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:24.890490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.2:53757)
0619 20:50:24.890624 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:24.890670 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260619 20:50:24.890718 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:24.890490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.3:36979)
0619 20:50:24.890623 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:24.890660 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260619 20:50:24.890952 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:24.890656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38413 (local address 127.0.98.1:56243)
0619 20:50:24.890839 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:24.890870 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:38413,127.0.98.61:42575,127.0.98.60:43969, num_attempts: 1)
I20260619 20:50:25.064754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 7236
I20260619 20:50:25.070269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 7463
I20260619 20:50:25.074843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 7599
I20260619 20:50:25.079456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 6956
I20260619 20:50:25.083879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 7143
2026-06-19T20:50:25Z chronyd exiting
I20260619 20:50:25.099537 392 test_util.cc:175] -----------------------------------------------
I20260619 20:50:25.099581 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781902166366061-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33303 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:50:25 dist-test-slave-0smx krb5kdc[7900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:50:25 dist-test-slave-0smx krb5kdc[7900](info): set up 2 sockets
Jun 19 20:50:25 dist-test-slave-0smx krb5kdc[7900](info): commencing operation
krb5kdc: starting...
W20260619 20:50:26.121697 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 20:50:26.232657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:26.311934 7913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:26.312167 7913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:26.312204 7913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:26.313442 7913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:26.313500 7913 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:50:26.313525 7913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:26.313583 7913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:26.313614 7913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:26.313642 7913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:26.315150 7913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37389
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:26.315412 7913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:26.315642 7913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:26.320598 7918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.320717 7919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:26.320613 7913 server_base.cc:1034] running on GCE node
W20260619 20:50:26.320613 7921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:26.321043 7913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:26.321467 7913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:26.322621 7913 hybrid_clock.cc:648] HybridClock initialized: now 1781902226322611 us; error 28 us; skew 500 ppm
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:26.324962 7913 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 20:50:26.325460 7913 webserver.cc:460] Webserver started at http://127.0.98.62:38557/ using document root <none> and password file <none>
I20260619 20:50:26.325642 7913 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:26.325681 7913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:26.325768 7913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:26.326730 7913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "4b50142f8de5406bba0bc96e2f9af314"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "cfcacdb32eb541b042c0bf6f54956cb3"
server_key_iv: "0b96cd73cdadf2b5834545025912e1ba"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.327018 7913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b50142f8de5406bba0bc96e2f9af314"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "cfcacdb32eb541b042c0bf6f54956cb3"
server_key_iv: "0b96cd73cdadf2b5834545025912e1ba"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.328249 7913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:50:26.328958 7928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.329107 7913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:26.329190 7913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "4b50142f8de5406bba0bc96e2f9af314"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "cfcacdb32eb541b042c0bf6f54956cb3"
server_key_iv: "0b96cd73cdadf2b5834545025912e1ba"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.329243 7913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:26.351616 7913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:26.353710 7913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:26.353819 7913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:26.358546 7913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37389
I20260619 20:50:26.358632 7990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37389 every 8 connection(s)
I20260619 20:50:26.358951 7913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:50:26.359642 7991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:26.361951 7991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Bootstrap starting.
I20260619 20:50:26.362534 7991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:26.362821 7991 log.cc:824] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:26.363471 7991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: No bootstrap required, opened a new log
I20260619 20:50:26.365558 7991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b50142f8de5406bba0bc96e2f9af314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260619 20:50:26.365713 7991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:26.365757 7991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b50142f8de5406bba0bc96e2f9af314, State: Initialized, Role: FOLLOWER
I20260619 20:50:26.365923 7991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b50142f8de5406bba0bc96e2f9af314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260619 20:50:26.366004 7991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:50:26.366055 7991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:50:26.366106 7991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:26.366611 7991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b50142f8de5406bba0bc96e2f9af314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260619 20:50:26.366725 7991 leader_election.cc:302] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b50142f8de5406bba0bc96e2f9af314; no voters:
I20260619 20:50:26.366901 7991 leader_election.cc:288] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:50:26.366940 7994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:26.367105 7994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [term 1 LEADER]: Becoming Leader. State: Replica: 4b50142f8de5406bba0bc96e2f9af314, State: Running, Role: LEADER
I20260619 20:50:26.367179 7991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:50:26.367389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 7913
I20260619 20:50:26.367488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:50:26.367655 392 external_mini_cluster.cc:1428] Setting key e5e0e799049f6b9a68ea95457ebf4699
I20260619 20:50:26.367278 7994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b50142f8de5406bba0bc96e2f9af314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } }
I20260619 20:50:26.368363 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b50142f8de5406bba0bc96e2f9af314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b50142f8de5406bba0bc96e2f9af314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } } }
I20260619 20:50:26.368459 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:26.368673 7996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b50142f8de5406bba0bc96e2f9af314. Latest consensus state: current_term: 1 leader_uuid: "4b50142f8de5406bba0bc96e2f9af314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b50142f8de5406bba0bc96e2f9af314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37389 } } }
I20260619 20:50:26.368922 7996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:26.368918 8006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:50:26.369529 8006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:50:26.371089 8006 catalog_manager.cc:1347] Generated new cluster ID: 9b56835ffd7f4ac1ba144278f32547c2
I20260619 20:50:26.371142 8006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:26.378997 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.368054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.0.98.62:37389)
0619 20:50:26.368334 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.368345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.368377 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:50:26.368492 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.368499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.368573 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.368733 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.369514 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.370941 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.371994 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.372601 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.372694 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.375354 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.375357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.375358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.375370 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.377221 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.377595 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.377597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.377598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.377698 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.377859 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.377859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.377859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.378127 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.378260 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.378638 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.378766 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":76,"threads_started":1}
I20260619 20:50:26.396770 8006 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:50:26.397406 8006 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:50:26.400892 8006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Generated new TSK 0
I20260619 20:50:26.401063 8006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 20:50:26.428439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:26.507220 8017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:26.507403 8017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:26.507426 8017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:26.508466 8017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:26.508498 8017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:26.508517 8017 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:50:26.508533 8017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:50:26.509948 8017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:26.510111 8017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:26.510286 8017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:26.515012 8022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.515080 8025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.515000 8023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:26.515339 8017 server_base.cc:1034] running on GCE node
I20260619 20:50:26.515532 8017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:26.515782 8017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:26.516934 8017 hybrid_clock.cc:648] HybridClock initialized: now 1781902226516919 us; error 27 us; skew 500 ppm
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:26.519286 8017 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:50:26.519707 8017 webserver.cc:460] Webserver started at http://127.0.98.1:36821/ using document root <none> and password file <none>
I20260619 20:50:26.519929 8017 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:26.519994 8017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:26.520138 8017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:26.521119 8017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "41aafc9f53dd48ec91204be2a663c417"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "c2b2d441246b5c2417720ab524ea90d7"
server_key_iv: "d5fe4bcefa554c5aa2d0d0b9a4b074be"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.521492 8017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "41aafc9f53dd48ec91204be2a663c417"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "c2b2d441246b5c2417720ab524ea90d7"
server_key_iv: "d5fe4bcefa554c5aa2d0d0b9a4b074be"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.522770 8017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:50:26.523447 8032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.523640 8017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:50:26.523728 8017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "41aafc9f53dd48ec91204be2a663c417"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "c2b2d441246b5c2417720ab524ea90d7"
server_key_iv: "d5fe4bcefa554c5aa2d0d0b9a4b074be"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.523813 8017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:26.559450 8017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:26.561407 8017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:26.561564 8017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:26.562074 8017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:26.562134 8017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.562176 8017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:26.562206 8017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.569576 8017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44751
I20260619 20:50:26.569643 8147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44751 every 8 connection(s)
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:26.569984 8017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:50:26.573045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8017
I20260619 20:50:26.573160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:50:26.573396 392 external_mini_cluster.cc:1428] Setting key e898fe6b0e41760e3d58209f0ec0bafd
I20260619 20:50:26.573899 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.562646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.1:56717)
0619 20:50:26.562815 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.562866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.563346 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.563599 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.563602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.563956 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:26.564256 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.564269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.565503 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.565504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.566891 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.566895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.567488 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.567489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.567640 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.568201 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:26.568211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:26.570359 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:26.572270 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.572276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.572278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.572597 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.572780 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.572781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.572782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.572932 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.573269 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:26.573275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:26.573478 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.573590 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.573743 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260619 20:50:26.574239 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.562657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56717 (local address 127.0.98.62:37389)
0619 20:50:26.562712 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.562714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.562970 (+ 256us) server_negotiation.cc:407] Connection header received
0619 20:50:26.563449 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.563450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.563463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.563534 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.564564 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.565394 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.566994 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.567346 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.567415 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.570466 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.570467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.570468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.570475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.572165 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.572665 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.572666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.572667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.572727 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.573016 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.573017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.573017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.573174 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.573238 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.574067 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.574142 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:50:26.574581 8148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37389
I20260619 20:50:26.574676 8148 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:26.574867 8148 heartbeater.cc:511] Master 127.0.98.62:37389 requested a full tablet report, sending...
I20260619 20:50:26.575387 7944 ts_manager.cc:194] Registered new tserver with Master: 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751)
I20260619 20:50:26.576295 7944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56717
I20260619 20:50:26.582417 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.575652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55035 (local address 127.0.98.62:37389)
0619 20:50:26.575688 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.575690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.576601 (+ 911us) server_negotiation.cc:407] Connection header received
0619 20:50:26.576687 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.576689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.576698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.576724 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.577049 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.577601 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.578725 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.578903 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.579017 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.579906 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.579907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.579907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.579913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.581281 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.581699 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.581700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.581700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.581746 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.581944 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.581944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.581945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.582041 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.582087 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.582260 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.582327 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:50:26.582448 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.575601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.1:55035)
0619 20:50:26.576544 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.576555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.576612 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.576779 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.576779 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.576926 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:26.576971 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.576972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.577690 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.577691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.578622 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.578624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.579027 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.579028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.579107 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.579500 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:26.579506 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:26.579829 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:26.581406 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.581407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.581407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.581624 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.581808 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.581809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.581809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.581876 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.582100 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:26.582100 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:26.582147 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.582219 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.582323 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 20:50:26.600147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:26.679838 8157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:26.680033 8157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:26.680056 8157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:26.681198 8157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:26.681239 8157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:26.681259 8157 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:50:26.681345 8157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:50:26.682813 8157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:26.683027 8157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:26.683251 8157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:26.688130 8165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.688187 8162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.688154 8163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:26.688306 8157 server_base.cc:1034] running on GCE node
I20260619 20:50:26.688612 8157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:26.688862 8157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:26.690013 8157 hybrid_clock.cc:648] HybridClock initialized: now 1781902226689997 us; error 28 us; skew 500 ppm
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:26.692293 8157 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:50:26.692739 8157 webserver.cc:460] Webserver started at http://127.0.98.2:33155/ using document root <none> and password file <none>
I20260619 20:50:26.692948 8157 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:26.693012 8157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:26.693152 8157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:26.694262 8157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "485082afe8a5c3b3a370fab012304c32"
server_key_iv: "314a6993a2d7c1a3ae11e93594e1926e"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.694628 8157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "485082afe8a5c3b3a370fab012304c32"
server_key_iv: "314a6993a2d7c1a3ae11e93594e1926e"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.695853 8157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:26.696492 8172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.696666 8157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:26.696756 8157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "485082afe8a5c3b3a370fab012304c32"
server_key_iv: "314a6993a2d7c1a3ae11e93594e1926e"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.696827 8157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:26.714154 8157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:26.715880 8157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:26.716030 8157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:26.716485 8157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:26.716542 8157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.716583 8157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:26.716621 8157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:26.723455 8157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43807
I20260619 20:50:26.723527 8287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43807 every 8 connection(s)
I20260619 20:50:26.723812 8157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:50:26.724570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8157
I20260619 20:50:26.724678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:50:26.724864 392 external_mini_cluster.cc:1428] Setting key 627aa885c28fe999895ad09a381a6618
I20260619 20:50:26.726723 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.716976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42979 (local address 127.0.98.62:37389)
0619 20:50:26.717042 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.717044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.717210 (+ 166us) server_negotiation.cc:407] Connection header received
0619 20:50:26.717757 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.717758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.717769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.717821 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.718454 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.719218 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.720836 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.721176 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.721247 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.723768 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.723770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.723770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.723777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.725275 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.725691 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.725692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.725692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.725747 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.726070 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.726071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.726071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.726191 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.726282 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.726552 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.726627 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:50:26.726897 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.716961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.2:42979)
0619 20:50:26.717095 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.717136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.717655 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.717884 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.717888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.718115 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:26.718368 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.718372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.719324 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.719325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.720731 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.720734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.721267 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.721268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.721454 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.721915 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:26.721926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:26.723655 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:26.725386 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.725392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.725394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.725599 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.725838 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.725838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.725839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.725988 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.726272 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:26.726278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:26.726419 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.726519 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.726669 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260619 20:50:26.727384 8288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37389
I20260619 20:50:26.727478 8288 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:26.727660 8288 heartbeater.cc:511] Master 127.0.98.62:37389 requested a full tablet report, sending...
I20260619 20:50:26.728457 7944 ts_manager.cc:194] Registered new tserver with Master: 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807)
I20260619 20:50:26.728840 7944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42979
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:50:26.734879 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.728967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42075 (local address 127.0.98.62:37389)
0619 20:50:26.729158 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.729160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.729168 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:26.729186 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.729187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.729197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.729252 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.729669 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.730431 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.731153 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.731353 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.731423 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.732337 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.732338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.732338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.732345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.733765 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.734121 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.734122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.734123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.734179 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.734387 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.734388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.734388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.734572 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.734636 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.734767 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.734822 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260619 20:50:26.734879 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.728401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.2:42075)
0619 20:50:26.728515 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.728525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.728571 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.729276 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.729277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.729489 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:26.729560 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.729562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.730546 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.730546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.731048 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.731050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.731435 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.731435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.731498 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.731917 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:26.731922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:26.732256 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:26.733854 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.733855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.733855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.734017 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.734245 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.734246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.734247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.734310 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.734650 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:26.734651 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:26.734688 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.734742 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.734812 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 20:50:26.749926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37389
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:26.828439 8297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:26.828619 8297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:26.828648 8297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:26.829917 8297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:26.829957 8297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:26.829978 8297 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:50:26.829995 8297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:50:26.831532 8297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:26.831753 8297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:26.831974 8297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:50:26.836941 8297 server_base.cc:1034] running on GCE node
W20260619 20:50:26.836903 8305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.836874 8303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:26.836884 8302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:26.837270 8297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:26.837600 8297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:26.838764 8297 hybrid_clock.cc:648] HybridClock initialized: now 1781902226838759 us; error 35 us; skew 500 ppm
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:26.841058 8297 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:50:26.841527 8297 webserver.cc:460] Webserver started at http://127.0.98.3:39515/ using document root <none> and password file <none>
I20260619 20:50:26.841748 8297 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:26.841814 8297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:26.841976 8297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:26.842948 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "5866b764233f42c094831eeefdceec6f"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "c1bbcfe08226f50f209d97caf280a7fd"
server_key_iv: "4ecb06570aa848671aa42e2bc32a418e"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.843331 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5866b764233f42c094831eeefdceec6f"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "c1bbcfe08226f50f209d97caf280a7fd"
server_key_iv: "4ecb06570aa848671aa42e2bc32a418e"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.844668 8297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:50:26.845356 8312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.845523 8297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:50:26.845628 8297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "5866b764233f42c094831eeefdceec6f"
format_stamp: "Formatted at 2026-06-19 20:50:26 on dist-test-slave-0smx"
server_key: "c1bbcfe08226f50f209d97caf280a7fd"
server_key_iv: "4ecb06570aa848671aa42e2bc32a418e"
server_key_version: "encryptionkey@0"
I20260619 20:50:26.845734 8297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:26.855247 8297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:26.856911 8297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:26.857059 8297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:26.857568 8297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:26.857625 8297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:26.857664 8297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:26.857709 8297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:26.864778 8297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36811
I20260619 20:50:26.864842 8427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36811 every 8 connection(s)
I20260619 20:50:26.865207 8297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:50:26.867609 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.858043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42151 (local address 127.0.98.62:37389)
0619 20:50:26.858109 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.858113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.858264 (+ 151us) server_negotiation.cc:407] Connection header received
0619 20:50:26.858736 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.858738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.858752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.858820 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.859584 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.860414 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.861970 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.862321 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.862407 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.864785 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.864786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.864787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.864794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.866228 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.866667 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.866668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.866669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.866728 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.866977 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.866978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.866978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.867093 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.867188 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.867446 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.867521 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:50:26.867806 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.858013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.3:42151)
0619 20:50:26.858147 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.858182 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.858646 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.858880 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.858883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.859216 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:26.859482 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.859487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.860517 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.860519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.861879 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.861881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.862436 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.862437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.862588 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.863170 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:26.863178 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:26.864696 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:26.866338 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.866344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.866346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.866552 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.866803 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.866804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.866804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.866909 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.867181 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:26.867187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:26.867319 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.867425 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.867570 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260619 20:50:26.868294 8428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37389
I20260619 20:50:26.868378 8428 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:26.868559 8428 heartbeater.cc:511] Master 127.0.98.62:37389 requested a full tablet report, sending...
I20260619 20:50:26.869009 7944 ts_manager.cc:194] Registered new tserver with Master: 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:26.869604 7944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42151
I20260619 20:50:26.874074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8297
I20260619 20:50:26.874200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:50:26.874250 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.869410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50385 (local address 127.0.98.62:37389)
0619 20:50:26.869463 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.869465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.869509 (+ 44us) server_negotiation.cc:407] Connection header received
0619 20:50:26.869582 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.869583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.869592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.869648 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.869983 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.870411 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.871042 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.871206 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.871253 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.871964 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.871965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.871965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.871970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.873374 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.873639 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.873639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.873640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.873680 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.873785 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.873785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.873785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.873864 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.873907 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.874068 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.874116 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:50:26.874300 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.869237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.98.3:50385)
0619 20:50:26.869453 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.869462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.869521 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.869650 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.869651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.869843 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:26.869906 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.869908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.870498 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.870499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.870975 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.870976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.871265 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.871266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.871307 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.871635 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:26.871638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:26.871880 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:26.873439 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.873440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.873440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.873582 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.873713 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:26.873714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:26.873714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:26.873753 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:26.873922 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:26.873922 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:26.873967 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.874054 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.874184 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260619 20:50:26.874413 392 external_mini_cluster.cc:1428] Setting key eb91e5caa80cdf250ab7bde0d8aa8dd7
I20260619 20:50:26.875450 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:50:26.895777 7999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:26.890464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.98.62:37389)
0619 20:50:26.890524 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.890526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.890536 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:50:26.890608 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.890609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.890625 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.890686 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.891262 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.891734 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.892534 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.892733 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.892834 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.895664 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45708: BlockingRecv error: recv got EOF from 127.0.0.1:45708 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 20:50:26 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:26.902679 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.896442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.98.62:37389)
0619 20:50:26.896491 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.896493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.896537 (+ 44us) server_negotiation.cc:407] Connection header received
0619 20:50:26.896582 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.896583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.896590 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.896641 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:26.897036 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.897521 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.898145 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.898316 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.898368 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.900273 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:26.900274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:26.900274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:26.900279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:26.901795 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.902091 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.902092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.902093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.902147 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:26.902313 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:26.902313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:26.902313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:26.902400 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:26.902468 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.902571 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.902622 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:50:26.904109 7945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:50:26.905673 7945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:50:26.912130 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.907979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36811 (local address 127.0.0.1:51986)
0619 20:50:26.908128 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.908188 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.908266 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.908507 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.908510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.908512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:26.908568 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.908570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.909710 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.909712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.910841 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.910844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.911697 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.911698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.911907 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.911922 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.912007 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.912060 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260619 20:50:26.912184 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.908063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.0.98.3:36811)
0619 20:50:26.908240 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.908245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.908257 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:50:26.908394 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.908396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.908411 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.908506 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:26.908671 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.909631 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.910924 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.911597 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.911694 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.911775 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.912042 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.912082 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260619 20:50:26.912463 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.908181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44751 (local address 127.0.0.1:39080)
0619 20:50:26.908558 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.908567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.908765 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.908945 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.908946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.908948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:26.908984 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.908985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.910209 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.910210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.911288 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.911290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.912216 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.912217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.912273 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.912276 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.912326 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.912355 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260619 20:50:26.912492 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.908415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.98.1:44751)
0619 20:50:26.908741 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.908745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.908756 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:50:26.908853 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.908855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.908871 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.908946 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:26.909049 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.910122 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.911371 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.912062 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.912123 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.912189 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.912345 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.912387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":69,"threads_started":1}
I20260619 20:50:26.912645 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.908246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.2:43807)
0619 20:50:26.908443 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:26.908448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:26.908593 (+ 145us) server_negotiation.cc:407] Connection header received
0619 20:50:26.908674 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:26.908676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:26.908691 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:26.908767 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:26.908950 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.910093 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.911244 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.911961 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.912031 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.912109 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:26.912481 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 20:50:26.912528 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260619 20:50:26.912397 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:26.908005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43807 (local address 127.0.0.1:55048)
0619 20:50:26.908489 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:26.908499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:26.908600 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:26.908798 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:26.908799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:26.908801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:26.908851 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.908852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.910193 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.910194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.911166 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:26.911168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:26.912157 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:26.912158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:26.912224 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:26.912228 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:26.912276 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:50:26.912318 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":30,"thread_start_us":79,"threads_started":1}
I20260619 20:50:26.913213 8362 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 (DEFAULT_TABLE table=test-table [id=ee7e7b0649134d32b4f9ec17489287f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:26.913503 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 (DEFAULT_TABLE table=test-table [id=ee7e7b0649134d32b4f9ec17489287f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:26.913602 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:26.913574 8222 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 (DEFAULT_TABLE table=test-table [id=ee7e7b0649134d32b4f9ec17489287f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:26.913738 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:26.913892 8222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:26.916543 8456 tablet_bootstrap.cc:492] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Bootstrap starting.
I20260619 20:50:26.917047 8456 tablet_bootstrap.cc:654] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:26.917259 8457 tablet_bootstrap.cc:492] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Bootstrap starting.
I20260619 20:50:26.917518 8456 log.cc:824] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:26.917781 8458 tablet_bootstrap.cc:492] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Bootstrap starting.
I20260619 20:50:26.917908 8457 tablet_bootstrap.cc:654] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:26.918184 8457 log.cc:824] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:26.918277 8456 tablet_bootstrap.cc:492] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: No bootstrap required, opened a new log
I20260619 20:50:26.918355 8458 tablet_bootstrap.cc:654] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:26.918350 8456 ts_tablet_manager.cc:1397] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:50:26.918704 8458 log.cc:824] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:26.918834 8457 tablet_bootstrap.cc:492] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: No bootstrap required, opened a new log
I20260619 20:50:26.918895 8457 ts_tablet_manager.cc:1397] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:50:26.919445 8458 tablet_bootstrap.cc:492] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: No bootstrap required, opened a new log
I20260619 20:50:26.919512 8458 ts_tablet_manager.cc:1397] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:50:26.920061 8456 raft_consensus.cc:348] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:26.920258 8456 raft_consensus.cc:374] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:26.920326 8456 raft_consensus.cc:729] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41aafc9f53dd48ec91204be2a663c417, State: Initialized, Role: FOLLOWER
I20260619 20:50:26.920457 8456 consensus_queue.cc:260] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:26.920805 8456 ts_tablet_manager.cc:1428] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:50:26.920810 8148 heartbeater.cc:503] Master 127.0.98.62:37389 was elected leader, sending a full tablet report...
I20260619 20:50:26.920852 8457 raft_consensus.cc:348] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:26.920992 8457 raft_consensus.cc:374] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:26.921022 8457 raft_consensus.cc:729] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5866b764233f42c094831eeefdceec6f, State: Initialized, Role: FOLLOWER
I20260619 20:50:26.921132 8457 consensus_queue.cc:260] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:26.921404 8457 ts_tablet_manager.cc:1428] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260619 20:50:26.921443 8428 heartbeater.cc:503] Master 127.0.98.62:37389 was elected leader, sending a full tablet report...
I20260619 20:50:26.921494 8458 raft_consensus.cc:348] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:26.921640 8458 raft_consensus.cc:374] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:26.921674 8458 raft_consensus.cc:729] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Initialized, Role: FOLLOWER
I20260619 20:50:26.921765 8458 consensus_queue.cc:260] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:26.922010 8458 ts_tablet_manager.cc:1428] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:50:26.922065 8288 heartbeater.cc:503] Master 127.0.98.62:37389 was elected leader, sending a full tablet report...
W20260619 20:50:26.974465 8289 tablet.cc:2369] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:50:27.073089 8149 tablet.cc:2369] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:50:27.104791 8464 raft_consensus.cc:482] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:50:27.104905 8464 raft_consensus.cc:504] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:27.105274 8464 leader_election.cc:288] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811), 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751)
I20260619 20:50:27.108150 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.105577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46747 (local address 127.0.98.3:36811)
0619 20:50:27.105634 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.105638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.105651 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:50:27.105680 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.105681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.105693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.105765 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:27.105949 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.106494 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.107442 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.107804 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.107857 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.107865 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.108037 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.108077 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:50:27.108183 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.105717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50183 (local address 127.0.98.1:44751)
0619 20:50:27.105767 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.105770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.105845 (+ 75us) server_negotiation.cc:407] Connection header received
0619 20:50:27.105895 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.105896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.105908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.105958 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:27.106119 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.106585 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.107482 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.107815 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.107904 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.107914 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.108067 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.108101 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:50:27.108213 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.105614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44751 (local address 127.0.98.2:50183)
0619 20:50:27.105767 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.105778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.105836 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.105970 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.105971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.105973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:27.106047 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.106048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.106659 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.106660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.107372 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.107374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.107916 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.107917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.107979 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.107982 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.108053 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.108094 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":13,"thread_start_us":97,"threads_started":1}
I20260619 20:50:27.108232 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.105490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36811 (local address 127.0.98.2:46747)
0619 20:50:27.105544 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.105557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.105601 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.105779 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.105780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.105782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:27.105857 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.105859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.106568 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.106568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.107356 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.107357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.107878 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.107879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.107960 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.107964 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.108016 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.108077 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260619 20:50:27.108537 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd99d5b1e8f47c09edeb9f0db3d48d9" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417" is_pre_election: true
I20260619 20:50:27.108544 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd99d5b1e8f47c09edeb9f0db3d48d9" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f" is_pre_election: true
I20260619 20:50:27.108700 8382 raft_consensus.cc:2454] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 0.
I20260619 20:50:27.108700 8091 raft_consensus.cc:2454] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 0.
I20260619 20:50:27.108920 8176 leader_election.cc:302] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aafc9f53dd48ec91204be2a663c417, 52ed3cae2d3e4f368bee5e2d3fd9a6ca; no voters:
I20260619 20:50:27.109032 8464 raft_consensus.cc:2781] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:50:27.109086 8464 raft_consensus.cc:482] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:50:27.109108 8464 raft_consensus.cc:3037] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.110085 8464 raft_consensus.cc:504] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:27.110211 8464 leader_election.cc:288] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 election: Requested vote from peers 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811), 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751)
I20260619 20:50:27.110363 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd99d5b1e8f47c09edeb9f0db3d48d9" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f"
I20260619 20:50:27.110388 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd99d5b1e8f47c09edeb9f0db3d48d9" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417"
I20260619 20:50:27.110437 8382 raft_consensus.cc:3037] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.110510 8091 raft_consensus.cc:3037] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.111145 8382 raft_consensus.cc:2454] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 1.
I20260619 20:50:27.111224 8091 raft_consensus.cc:2454] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 1.
I20260619 20:50:27.111313 8176 leader_election.cc:302] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, 5866b764233f42c094831eeefdceec6f; no voters:
I20260619 20:50:27.111444 8464 raft_consensus.cc:2781] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:27.111568 8464 raft_consensus.cc:686] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 LEADER]: Becoming Leader. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Running, Role: LEADER
I20260619 20:50:27.111671 8464 consensus_queue.cc:237] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } }
I20260619 20:50:27.112490 7944 catalog_manager.cc:5495] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } health_report { overall_health: HEALTHY } } }
W20260619 20:50:27.115736 8429 tablet.cc:2369] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:50:27.140836 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.138798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.0.98.2:43807)
0619 20:50:27.138841 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.138842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.138850 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:27.138904 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.138904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.138923 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.138980 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:27.139094 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.139539 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.140228 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.140401 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.140450 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.140579 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.140678 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.140735 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.140768 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.140783 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:50:27.142637 8382 raft_consensus.cc:1261] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Refusing update from remote peer 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.142655 8091 raft_consensus.cc:1261] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Refusing update from remote peer 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.142997 8464 consensus_queue.cc:1035] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.143129 8466 consensus_queue.cc:1035] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.146481 8470 mvcc.cc:205] Tried to move back new op lower bound from 7298671522373697536 to 7298671522250072064. Current Snapshot: MvccSnapshot[applied={T|T < 7298671522373697536 or (T in {7298671522373697536})}]
I20260619 20:50:27.147671 7945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:27.147764 7945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:27.148460 7945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Sending DeleteTablet for 3 replicas of tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9
I20260619 20:50:27.148794 8222 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260619 20:50:27.148804 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39080
I20260619 20:50:27.148895 8362 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51986
I20260619 20:50:27.148981 8480 tablet_replica.cc:330] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: stopping tablet replica
I20260619 20:50:27.149092 8480 raft_consensus.cc:2229] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:50:27.149092 7945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 20:50:27.149204 8480 raft_consensus.cc:2258] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:50:27.149246 7945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:50:27.149381 8479 tablet_replica.cc:330] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: stopping tablet replica
I20260619 20:50:27.149523 8479 raft_consensus.cc:2229] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:27.149718 8481 tablet_replica.cc:330] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: stopping tablet replica
I20260619 20:50:27.149981 8481 raft_consensus.cc:2229] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:27.149991 8480 ts_tablet_manager.cc:1905] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.150559 8362 tablet_service.cc:1467] Processing CreateTablet for tablet 7a7c4a01df744f919dc52e5d9434bf5e (DEFAULT_TABLE table=test-table [id=3c17a7bafec2427ebe0befee4d57e12c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.150606 8222 tablet_service.cc:1467] Processing CreateTablet for tablet 7a7c4a01df744f919dc52e5d9434bf5e (DEFAULT_TABLE table=test-table [id=3c17a7bafec2427ebe0befee4d57e12c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.150676 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a7c4a01df744f919dc52e5d9434bf5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.150691 8222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a7c4a01df744f919dc52e5d9434bf5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.150537 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 7a7c4a01df744f919dc52e5d9434bf5e (DEFAULT_TABLE table=test-table [id=3c17a7bafec2427ebe0befee4d57e12c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.151597 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a7c4a01df744f919dc52e5d9434bf5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.151597 8479 raft_consensus.cc:2258] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.151861 8480 ts_tablet_manager.cc:1918] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:27.151954 8457 tablet_bootstrap.cc:492] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Bootstrap starting.
I20260619 20:50:27.152076 8481 raft_consensus.cc:2258] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.152343 8457 tablet_bootstrap.cc:654] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.152706 8480 log.cc:1196] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/8cd99d5b1e8f47c09edeb9f0db3d48d9
I20260619 20:50:27.152982 8457 tablet_bootstrap.cc:492] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: No bootstrap required, opened a new log
I20260619 20:50:27.153034 8457 ts_tablet_manager.cc:1397] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:50:27.153077 8456 tablet_bootstrap.cc:492] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Bootstrap starting.
I20260619 20:50:27.153081 8458 tablet_bootstrap.cc:492] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Bootstrap starting.
I20260619 20:50:27.153174 8457 raft_consensus.cc:348] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.153247 8457 raft_consensus.cc:374] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.153271 8457 raft_consensus.cc:729] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5866b764233f42c094831eeefdceec6f, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.153342 8457 consensus_queue.cc:260] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.153426 8457 ts_tablet_manager.cc:1428] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.153460 8456 tablet_bootstrap.cc:654] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.153520 8458 tablet_bootstrap.cc:654] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.154114 8481 ts_tablet_manager.cc:1905] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.154392 8480 ts_tablet_manager.cc:1939] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting consensus metadata
I20260619 20:50:27.154740 7930 catalog_manager.cc:4841] TS 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807): tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 (table test-table [id=ee7e7b0649134d32b4f9ec17489287f3]) successfully deleted
I20260619 20:50:27.154852 8479 ts_tablet_manager.cc:1905] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.155503 8481 ts_tablet_manager.cc:1918] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:27.155558 8481 log.cc:1196] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/8cd99d5b1e8f47c09edeb9f0db3d48d9
I20260619 20:50:27.155803 8481 ts_tablet_manager.cc:1939] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 5866b764233f42c094831eeefdceec6f: Deleting consensus metadata
I20260619 20:50:27.155925 8479 ts_tablet_manager.cc:1918] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:27.156014 8479 log.cc:1196] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/8cd99d5b1e8f47c09edeb9f0db3d48d9
I20260619 20:50:27.156292 7932 catalog_manager.cc:4841] TS 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811): tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 (table test-table [id=ee7e7b0649134d32b4f9ec17489287f3]) successfully deleted
I20260619 20:50:27.156378 8479 ts_tablet_manager.cc:1939] T 8cd99d5b1e8f47c09edeb9f0db3d48d9 P 41aafc9f53dd48ec91204be2a663c417: Deleting consensus metadata
I20260619 20:50:27.156443 8456 tablet_bootstrap.cc:492] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: No bootstrap required, opened a new log
I20260619 20:50:27.156495 8456 ts_tablet_manager.cc:1397] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:50:27.156631 8456 raft_consensus.cc:348] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.156698 8456 raft_consensus.cc:374] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.156726 8456 raft_consensus.cc:729] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41aafc9f53dd48ec91204be2a663c417, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.156754 7932 catalog_manager.cc:4841] TS 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751): tablet 8cd99d5b1e8f47c09edeb9f0db3d48d9 (table test-table [id=ee7e7b0649134d32b4f9ec17489287f3]) successfully deleted
I20260619 20:50:27.156764 8456 consensus_queue.cc:260] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.156862 8456 ts_tablet_manager.cc:1428] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.157284 8458 tablet_bootstrap.cc:492] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: No bootstrap required, opened a new log
I20260619 20:50:27.157357 8458 ts_tablet_manager.cc:1397] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 20:50:27.157509 8458 raft_consensus.cc:348] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.157572 8458 raft_consensus.cc:374] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.157604 8458 raft_consensus.cc:729] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.157660 8458 consensus_queue.cc:260] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.157742 8458 ts_tablet_manager.cc:1428] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.199462 8463 raft_consensus.cc:482] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:50:27.199568 8463 raft_consensus.cc:504] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.199893 8463 leader_election.cc:288] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751), 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807)
I20260619 20:50:27.203008 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.200132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36977 (local address 127.0.98.1:44751)
0619 20:50:27.200283 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.200285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.200291 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:50:27.200313 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.200314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.200321 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.200379 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:27.200550 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.201032 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.202084 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.202635 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.202711 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.202722 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.202901 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.202936 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 20:50:27.203024 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.200057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44751 (local address 127.0.98.3:36977)
0619 20:50:27.200138 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.200150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.200192 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.200395 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.200396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.200398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:27.200470 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.200472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.201138 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.201139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.201982 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.201984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.202729 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.202730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.202810 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.202814 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.202889 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.202936 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 20:50:27.203706 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c4a01df744f919dc52e5d9434bf5e" candidate_uuid: "5866b764233f42c094831eeefdceec6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417" is_pre_election: true
I20260619 20:50:27.203806 8091 raft_consensus.cc:2454] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5866b764233f42c094831eeefdceec6f in term 0.
I20260619 20:50:27.203999 8316 leader_election.cc:302] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aafc9f53dd48ec91204be2a663c417, 5866b764233f42c094831eeefdceec6f; no voters:
I20260619 20:50:27.204138 8463 raft_consensus.cc:2781] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:50:27.204175 8463 raft_consensus.cc:482] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:50:27.204205 8463 raft_consensus.cc:3037] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.204239 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.200392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57509 (local address 127.0.98.2:43807)
0619 20:50:27.200439 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.200441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.200447 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:50:27.200472 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.200473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.200480 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.200528 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:27.201339 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.202072 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.203318 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.203859 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.203907 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.203918 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.204141 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.204177 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:50:27.204428 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.200068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43807 (local address 127.0.98.3:57509)
0619 20:50:27.200220 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.200231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.200291 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.201159 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.201160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.201162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:27.201239 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.201240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.202156 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.202157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.203227 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.203230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.203977 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.203978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.204053 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.204056 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.204312 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.204352 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260619 20:50:27.204674 8242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c4a01df744f919dc52e5d9434bf5e" candidate_uuid: "5866b764233f42c094831eeefdceec6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" is_pre_election: true
I20260619 20:50:27.204804 8242 raft_consensus.cc:2454] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5866b764233f42c094831eeefdceec6f in term 0.
I20260619 20:50:27.205147 8463 raft_consensus.cc:504] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.205309 8463 leader_election.cc:288] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [CANDIDATE]: Term 1 election: Requested vote from peers 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751), 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807)
I20260619 20:50:27.205461 8242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c4a01df744f919dc52e5d9434bf5e" candidate_uuid: "5866b764233f42c094831eeefdceec6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca"
I20260619 20:50:27.205550 8242 raft_consensus.cc:3037] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.206125 8242 raft_consensus.cc:2454] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5866b764233f42c094831eeefdceec6f in term 1.
I20260619 20:50:27.206240 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c4a01df744f919dc52e5d9434bf5e" candidate_uuid: "5866b764233f42c094831eeefdceec6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417"
I20260619 20:50:27.206314 8091 raft_consensus.cc:3037] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.206321 8314 leader_election.cc:302] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, 5866b764233f42c094831eeefdceec6f; no voters:
I20260619 20:50:27.206416 8463 raft_consensus.cc:2781] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:27.206527 8463 raft_consensus.cc:686] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 1 LEADER]: Becoming Leader. State: Replica: 5866b764233f42c094831eeefdceec6f, State: Running, Role: LEADER
I20260619 20:50:27.206590 8463 consensus_queue.cc:237] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.207058 8091 raft_consensus.cc:2454] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5866b764233f42c094831eeefdceec6f in term 1.
I20260619 20:50:27.207221 7945 catalog_manager.cc:5495] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5866b764233f42c094831eeefdceec6f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5866b764233f42c094831eeefdceec6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } health_report { overall_health: HEALTHY } } }
Jun 19 20:50:27 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:27.233757 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.225986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37389 (local address 127.0.0.1:45728)
0619 20:50:27.226175 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.226191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.226282 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.226402 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.226403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.226710 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:27.226751 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.226753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.227503 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.227505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.228026 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.228027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.228443 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.228444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.228497 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.229049 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:27.229055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:27.230756 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:27.232544 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:27.232550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:27.232552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:27.232795 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:27.233024 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:27.233025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:27.233026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:27.233090 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:27.233389 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:27.233394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:27.233436 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.233489 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.233568 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260619 20:50:27.233764 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.226036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.0.98.62:37389)
0619 20:50:27.226121 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.226123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.226248 (+ 125us) server_negotiation.cc:407] Connection header received
0619 20:50:27.226331 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.226332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.226342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.226398 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:27.226840 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.227401 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.228178 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.228360 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.228443 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.230853 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:27.230855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:27.230855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:27.230863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:27.232457 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:27.232875 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:27.232877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:27.232877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:27.232939 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:27.233161 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:27.233162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:27.233162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:27.233320 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:27.233398 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.233542 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.233646 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:50:27.236058 7945 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45728:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:50:27.236279 7945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:50:27.237743 8222 tablet_service.cc:1467] Processing CreateTablet for tablet d4c68b04241e43839675e2ca854ef314 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=629e49b7d3b34b63990127256c8ae948]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:50:27.237756 8362 tablet_service.cc:1467] Processing CreateTablet for tablet d4c68b04241e43839675e2ca854ef314 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=629e49b7d3b34b63990127256c8ae948]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:50:27.237766 8075 tablet_service.cc:1467] Processing CreateTablet for tablet d4c68b04241e43839675e2ca854ef314 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=629e49b7d3b34b63990127256c8ae948]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:50:27.237896 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c68b04241e43839675e2ca854ef314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.237896 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c68b04241e43839675e2ca854ef314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.237906 8222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c68b04241e43839675e2ca854ef314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.239351 8457 tablet_bootstrap.cc:492] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: Bootstrap starting.
I20260619 20:50:27.239351 8456 tablet_bootstrap.cc:492] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: Bootstrap starting.
I20260619 20:50:27.239398 8458 tablet_bootstrap.cc:492] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Bootstrap starting.
I20260619 20:50:27.239773 8456 tablet_bootstrap.cc:654] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.239773 8457 tablet_bootstrap.cc:654] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.239795 8458 tablet_bootstrap.cc:654] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.240576 8456 tablet_bootstrap.cc:492] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: No bootstrap required, opened a new log
I20260619 20:50:27.240660 8456 ts_tablet_manager.cc:1397] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:27.240818 8456 raft_consensus.cc:348] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.240895 8456 raft_consensus.cc:374] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.240924 8456 raft_consensus.cc:729] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41aafc9f53dd48ec91204be2a663c417, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.240965 8456 consensus_queue.cc:260] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.241063 8456 ts_tablet_manager.cc:1428] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.241079 8458 tablet_bootstrap.cc:492] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: No bootstrap required, opened a new log
I20260619 20:50:27.241125 8458 ts_tablet_manager.cc:1397] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:50:27.241127 8462 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.241204 8462 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.241250 8458 raft_consensus.cc:348] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.241343 8458 raft_consensus.cc:374] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.241364 8458 raft_consensus.cc:729] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.241403 8458 consensus_queue.cc:260] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.241479 8458 ts_tablet_manager.cc:1428] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.241593 8475 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.241667 8475 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.241814 8457 tablet_bootstrap.cc:492] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: No bootstrap required, opened a new log
I20260619 20:50:27.241858 8457 ts_tablet_manager.cc:1397] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 20:50:27.242182 8457 raft_consensus.cc:348] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.242264 8457 raft_consensus.cc:374] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.242287 8457 raft_consensus.cc:729] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5866b764233f42c094831eeefdceec6f, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.242337 8457 consensus_queue.cc:260] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.242432 8457 ts_tablet_manager.cc:1428] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:50:27.242479 8463 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.242565 8463 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.536579 8475 raft_consensus.cc:482] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:50:27.536670 8475 raft_consensus.cc:504] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.536857 8475 leader_election.cc:288] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751), 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:27.537082 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c68b04241e43839675e2ca854ef314" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417" is_pre_election: true
I20260619 20:50:27.537200 8091 raft_consensus.cc:2454] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 0.
I20260619 20:50:27.537178 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c68b04241e43839675e2ca854ef314" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f" is_pre_election: true
I20260619 20:50:27.537339 8382 raft_consensus.cc:2454] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 0.
I20260619 20:50:27.537372 8176 leader_election.cc:302] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aafc9f53dd48ec91204be2a663c417, 52ed3cae2d3e4f368bee5e2d3fd9a6ca; no voters:
I20260619 20:50:27.537467 8475 raft_consensus.cc:2781] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:50:27.537519 8475 raft_consensus.cc:482] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:50:27.537534 8475 raft_consensus.cc:3037] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.538185 8475 raft_consensus.cc:504] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.538322 8475 leader_election.cc:288] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 election: Requested vote from peers 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751), 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:27.538509 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c68b04241e43839675e2ca854ef314" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f"
I20260619 20:50:27.538509 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c68b04241e43839675e2ca854ef314" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417"
I20260619 20:50:27.538579 8382 raft_consensus.cc:3037] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.538580 8091 raft_consensus.cc:3037] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.539296 8382 raft_consensus.cc:2454] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 1.
I20260619 20:50:27.539296 8091 raft_consensus.cc:2454] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 1.
I20260619 20:50:27.539507 8176 leader_election.cc:302] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, 5866b764233f42c094831eeefdceec6f; no voters:
I20260619 20:50:27.539599 8475 raft_consensus.cc:2781] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:27.539652 8475 raft_consensus.cc:686] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 LEADER]: Becoming Leader. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Running, Role: LEADER
I20260619 20:50:27.539696 8475 consensus_queue.cc:237] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.539922 8464 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: TxnStatusTablet state changed. Reason: New leader 52ed3cae2d3e4f368bee5e2d3fd9a6ca. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.539986 8464 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:50:27.540112 8498 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:50:27.540256 8498 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:50:27.540293 7945 catalog_manager.cc:5495] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } health_report { overall_health: UNKNOWN } } }
I20260619 20:50:27.597090 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.594674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.98.2:43807)
0619 20:50:27.594748 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.594750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.594773 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:50:27.594876 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.594877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.594888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.594945 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:27.595080 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.595599 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.596338 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.596597 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.596665 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.596831 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.596865 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.596931 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.596989 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.597009 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:50:27.597096 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.594619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43807 (local address 127.0.0.1:55060)
0619 20:50:27.594716 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.594739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.594793 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.594950 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.594951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.594952 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:50:27.595000 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.595003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.595690 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.595691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.596251 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.596253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.596666 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.596667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.596730 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.596750 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.596932 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.596940 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.596986 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.597033 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 20:50:27.598193 8091 raft_consensus.cc:1261] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Refusing update from remote peer 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.598201 8382 raft_consensus.cc:1261] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Refusing update from remote peer 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.598394 8464 consensus_queue.cc:1035] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.598507 8475 consensus_queue.cc:1035] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.599491 8463 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: TxnStatusTablet state changed. Reason: New leader 52ed3cae2d3e4f368bee5e2d3fd9a6ca. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.599579 8463 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.599660 8462 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: TxnStatusTablet state changed. Reason: New leader 52ed3cae2d3e4f368bee5e2d3fd9a6ca. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.599747 8462 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.599969 8469 mvcc.cc:205] Tried to move back new op lower bound from 7298671524240633856 to 7298671524003004416. Current Snapshot: MvccSnapshot[applied={T|T < 7298671524240633856}]
I20260619 20:50:27.600045 8466 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.600126 8466 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:50:27.600176 8466 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.600215 8466 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:50:27.600769 8463 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.600852 8463 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 5866b764233f42c094831eeefdceec6f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.600975 8471 mvcc.cc:205] Tried to move back new op lower bound from 7298671524240633856 to 7298671524003004416. Current Snapshot: MvccSnapshot[applied={T|T < 7298671524240633856}]
I20260619 20:50:27.601089 8462 tablet_replica.cc:439] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } } }
I20260619 20:50:27.601176 8462 tablet_replica.cc:442] T d4c68b04241e43839675e2ca854ef314 P 41aafc9f53dd48ec91204be2a663c417: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:50:27.604390 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.601876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.0.98.3:36811)
0619 20:50:27.601911 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.601912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.601921 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:50:27.602005 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.602005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.602024 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.602104 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:27.602177 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.602852 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.603559 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.603887 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.603972 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.604072 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.604194 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.604278 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.604303 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.604325 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:50:27.609091 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.606598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49769 (local address 127.0.98.2:43807)
0619 20:50:27.606635 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.606637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.606704 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:50:27.606765 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.606766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.606774 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.606815 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:27.606962 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.607363 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.608380 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.608633 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.608684 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.608894 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.608917 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.608949 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.609034 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.609048 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:50:27.609154 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.606485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43807 (local address 127.0.98.3:49769)
0619 20:50:27.606640 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.606651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.606712 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.606834 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.606835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.606844 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:50:27.606901 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.606903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.607438 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.607439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.608295 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.608298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.608732 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.608733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.608794 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.608807 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.608973 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.608975 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.609033 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.609075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260619 20:50:27.611495 8091 raft_consensus.cc:1261] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Refusing update from remote peer 5866b764233f42c094831eeefdceec6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.611513 8242 raft_consensus.cc:1261] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Refusing update from remote peer 5866b764233f42c094831eeefdceec6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.611775 8486 consensus_queue.cc:1035] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.611937 8486 consensus_queue.cc:1035] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.619989 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.617221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50391 (local address 127.0.98.3:36811)
0619 20:50:27.617264 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.617266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.617405 (+ 139us) server_negotiation.cc:407] Connection header received
0619 20:50:27.617460 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.617461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.617472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.617525 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:27.617683 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.618415 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.619180 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.619462 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.619546 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.619712 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.619765 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.619843 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.619905 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.619924 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:50:27.620021 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.617128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36811 (local address 127.0.98.2:50391)
0619 20:50:27.617325 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:27.617354 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:27.617411 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:27.617527 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:27.617528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:27.617529 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:50:27.617593 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.617594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.618493 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.618494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.619053 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:27.619054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.619542 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.619543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:27.619619 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.619634 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.619846 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.619848 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:27.619901 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:50:27.619946 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260619 20:50:27.629449 7944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:27.629580 7944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:27.630183 7944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Sending DeleteTablet for 3 replicas of tablet 7a7c4a01df744f919dc52e5d9434bf5e
I20260619 20:50:27.630463 8222 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a7c4a01df744f919dc52e5d9434bf5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260619 20:50:27.630568 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a7c4a01df744f919dc52e5d9434bf5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39080
I20260619 20:50:27.630661 8480 tablet_replica.cc:330] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: stopping tablet replica
I20260619 20:50:27.630720 8480 raft_consensus.cc:2229] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:27.630764 8480 raft_consensus.cc:2258] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.630836 8479 tablet_replica.cc:330] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: stopping tablet replica
I20260619 20:50:27.630885 8479 raft_consensus.cc:2229] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:27.630929 8479 raft_consensus.cc:2258] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.631201 8362 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a7c4a01df744f919dc52e5d9434bf5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51986
I20260619 20:50:27.631287 8481 tablet_replica.cc:330] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: stopping tablet replica
I20260619 20:50:27.631316 8479 ts_tablet_manager.cc:1905] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.631332 8481 raft_consensus.cc:2229] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:50:27.631414 8481 raft_consensus.cc:2258] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.631806 8481 ts_tablet_manager.cc:1905] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.633143 8479 ts_tablet_manager.cc:1918] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:50:27.633224 8479 log.cc:1196] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/7a7c4a01df744f919dc52e5d9434bf5e
I20260619 20:50:27.633488 8481 ts_tablet_manager.cc:1918] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:50:27.633505 8479 ts_tablet_manager.cc:1939] T 7a7c4a01df744f919dc52e5d9434bf5e P 41aafc9f53dd48ec91204be2a663c417: Deleting consensus metadata
I20260619 20:50:27.633566 8481 log.cc:1196] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/7a7c4a01df744f919dc52e5d9434bf5e
I20260619 20:50:27.633785 8481 ts_tablet_manager.cc:1939] T 7a7c4a01df744f919dc52e5d9434bf5e P 5866b764233f42c094831eeefdceec6f: Deleting consensus metadata
I20260619 20:50:27.633816 7932 catalog_manager.cc:4841] TS 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751): tablet 7a7c4a01df744f919dc52e5d9434bf5e (table test-table [id=3c17a7bafec2427ebe0befee4d57e12c]) successfully deleted
I20260619 20:50:27.634065 7932 catalog_manager.cc:4841] TS 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811): tablet 7a7c4a01df744f919dc52e5d9434bf5e (table test-table [id=3c17a7bafec2427ebe0befee4d57e12c]) successfully deleted
I20260619 20:50:27.634400 8480 ts_tablet_manager.cc:1905] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.636981 8480 ts_tablet_manager.cc:1918] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:50:27.637044 8480 log.cc:1196] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/7a7c4a01df744f919dc52e5d9434bf5e
I20260619 20:50:27.637265 8480 ts_tablet_manager.cc:1939] T 7a7c4a01df744f919dc52e5d9434bf5e P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting consensus metadata
I20260619 20:50:27.637615 7930 catalog_manager.cc:4841] TS 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807): tablet 7a7c4a01df744f919dc52e5d9434bf5e (table test-table [id=3c17a7bafec2427ebe0befee4d57e12c]) successfully deleted
Jun 19 20:50:27 dist-test-slave-0smx krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:50:27 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:27 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:50:27.658408 7999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:27.652772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.0.98.62:37389)
0619 20:50:27.652826 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.652829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.652976 (+ 147us) server_negotiation.cc:407] Connection header received
0619 20:50:27.653051 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.653053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.653064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.653139 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:27.653832 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.654347 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.655116 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.655411 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.655506 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.658350 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45730: BlockingRecv error: recv got EOF from 127.0.0.1:45730 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 20:50:27 dist-test-slave-0smx krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 20:50:27.666657 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.659564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.0.98.62:37389)
0619 20:50:27.659602 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.659604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.659763 (+ 159us) server_negotiation.cc:407] Connection header received
0619 20:50:27.659832 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.659834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.659859 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.659906 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:27.660338 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.660820 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.661593 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.661797 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.661882 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.664026 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:27.664028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:27.664028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:27.664036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:27.665550 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:27.665906 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:27.665907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:27.665908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:27.665965 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:27.666166 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:27.666167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:27.666167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:27.666272 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:27.666367 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.666477 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.666549 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:50:27.669216 7944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:50:27.669550 7944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:50:27.670924 8075 tablet_service.cc:1467] Processing CreateTablet for tablet dfecca8078c441fb8b1882bcb259dc2d (DEFAULT_TABLE table=test-table [id=1e2ba1ac267b46e9a014fbc303f3a740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.670924 8222 tablet_service.cc:1467] Processing CreateTablet for tablet dfecca8078c441fb8b1882bcb259dc2d (DEFAULT_TABLE table=test-table [id=1e2ba1ac267b46e9a014fbc303f3a740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.671065 8222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfecca8078c441fb8b1882bcb259dc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.671065 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfecca8078c441fb8b1882bcb259dc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.672328 8458 tablet_bootstrap.cc:492] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Bootstrap starting.
I20260619 20:50:27.672372 8362 tablet_service.cc:1467] Processing CreateTablet for tablet dfecca8078c441fb8b1882bcb259dc2d (DEFAULT_TABLE table=test-table [id=1e2ba1ac267b46e9a014fbc303f3a740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.672508 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfecca8078c441fb8b1882bcb259dc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.672667 8458 tablet_bootstrap.cc:654] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.673158 8458 tablet_bootstrap.cc:492] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: No bootstrap required, opened a new log
I20260619 20:50:27.673202 8458 ts_tablet_manager.cc:1397] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:27.673311 8458 raft_consensus.cc:348] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.673395 8458 raft_consensus.cc:374] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.673414 8458 raft_consensus.cc:729] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.673452 8458 consensus_queue.cc:260] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.673537 8458 ts_tablet_manager.cc:1428] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.673885 8457 tablet_bootstrap.cc:492] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Bootstrap starting.
I20260619 20:50:27.673919 8456 tablet_bootstrap.cc:492] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Bootstrap starting.
I20260619 20:50:27.674348 8456 tablet_bootstrap.cc:654] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.674348 8457 tablet_bootstrap.cc:654] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.675594 8456 tablet_bootstrap.cc:492] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: No bootstrap required, opened a new log
I20260619 20:50:27.675645 8456 ts_tablet_manager.cc:1397] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:50:27.675750 8457 tablet_bootstrap.cc:492] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: No bootstrap required, opened a new log
I20260619 20:50:27.675787 8457 ts_tablet_manager.cc:1397] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:50:27.675813 8456 raft_consensus.cc:348] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.675890 8456 raft_consensus.cc:374] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.675909 8456 raft_consensus.cc:729] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41aafc9f53dd48ec91204be2a663c417, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.675958 8456 consensus_queue.cc:260] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.675946 8457 raft_consensus.cc:348] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.676005 8457 raft_consensus.cc:374] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.676026 8457 raft_consensus.cc:729] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5866b764233f42c094831eeefdceec6f, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.676043 8456 ts_tablet_manager.cc:1428] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.676064 8457 consensus_queue.cc:260] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.676160 8457 ts_tablet_manager.cc:1428] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.900404 8466 raft_consensus.cc:482] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:50:27.900532 8466 raft_consensus.cc:504] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.900720 8466 leader_election.cc:288] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751), 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:27.900945 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfecca8078c441fb8b1882bcb259dc2d" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f" is_pre_election: true
I20260619 20:50:27.900961 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfecca8078c441fb8b1882bcb259dc2d" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417" is_pre_election: true
I20260619 20:50:27.901065 8382 raft_consensus.cc:2454] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 0.
I20260619 20:50:27.901103 8091 raft_consensus.cc:2454] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 0.
I20260619 20:50:27.901223 8176 leader_election.cc:302] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, 5866b764233f42c094831eeefdceec6f; no voters:
I20260619 20:50:27.901329 8466 raft_consensus.cc:2781] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:50:27.901376 8466 raft_consensus.cc:482] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:50:27.901408 8466 raft_consensus.cc:3037] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.902135 8466 raft_consensus.cc:504] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.902266 8466 leader_election.cc:288] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 election: Requested vote from peers 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751), 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:27.902413 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfecca8078c441fb8b1882bcb259dc2d" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41aafc9f53dd48ec91204be2a663c417"
I20260619 20:50:27.902451 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfecca8078c441fb8b1882bcb259dc2d" candidate_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f"
I20260619 20:50:27.902491 8091 raft_consensus.cc:3037] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.902521 8382 raft_consensus.cc:3037] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:27.903155 8382 raft_consensus.cc:2454] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 1.
I20260619 20:50:27.903208 8091 raft_consensus.cc:2454] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ed3cae2d3e4f368bee5e2d3fd9a6ca in term 1.
I20260619 20:50:27.903362 8176 leader_election.cc:302] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aafc9f53dd48ec91204be2a663c417, 52ed3cae2d3e4f368bee5e2d3fd9a6ca; no voters:
I20260619 20:50:27.903492 8466 raft_consensus.cc:2781] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:27.903571 8466 raft_consensus.cc:686] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 LEADER]: Becoming Leader. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Running, Role: LEADER
I20260619 20:50:27.903626 8466 consensus_queue.cc:237] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.904203 7945 catalog_manager.cc:5495] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } health_report { overall_health: UNKNOWN } } }
I20260619 20:50:27.960876 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:27.958324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.0.98.2:43807)
0619 20:50:27.958397 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:27.958398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:27.958406 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:27.958430 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:27.958431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:27.958440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:27.958514 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:27.958678 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.959226 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.960124 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:27.960339 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:27.960412 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:27.960601 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:27.960640 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:27.960707 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:27.960772 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 20:50:27.960792 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:50:27.961766 8382 raft_consensus.cc:1261] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Refusing update from remote peer 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.961809 8091 raft_consensus.cc:1261] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Refusing update from remote peer 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:27.962051 8466 consensus_queue.cc:1035] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.962167 8464 consensus_queue.cc:1035] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:27.964515 7945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:27.964627 7945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:27.965198 7945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Sending DeleteTablet for 3 replicas of tablet dfecca8078c441fb8b1882bcb259dc2d
I20260619 20:50:27.965440 8222 tablet_service.cc:1514] Processing DeleteTablet for tablet dfecca8078c441fb8b1882bcb259dc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260619 20:50:27.965440 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet dfecca8078c441fb8b1882bcb259dc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39080
I20260619 20:50:27.965523 8480 tablet_replica.cc:330] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: stopping tablet replica
I20260619 20:50:27.965502 8362 tablet_service.cc:1514] Processing DeleteTablet for tablet dfecca8078c441fb8b1882bcb259dc2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51986
I20260619 20:50:27.965523 8479 tablet_replica.cc:330] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: stopping tablet replica
I20260619 20:50:27.965569 8480 raft_consensus.cc:2229] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:50:27.965576 8479 raft_consensus.cc:2229] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:27.965584 8481 tablet_replica.cc:330] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: stopping tablet replica
I20260619 20:50:27.965617 8479 raft_consensus.cc:2258] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.965626 8480 raft_consensus.cc:2258] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.965632 8481 raft_consensus.cc:2229] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:27.965677 8481 raft_consensus.cc:2258] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:27.965691 7945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:50:27.965824 7945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:50:27.966005 8479 ts_tablet_manager.cc:1905] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.966151 8480 ts_tablet_manager.cc:1905] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.966974 8075 tablet_service.cc:1467] Processing CreateTablet for tablet ddc6fd5142f04f7b8feb912dbfe38b5c (DEFAULT_TABLE table=test-table [id=714860cc3f1745058a7704c36e9ea5cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.966976 8222 tablet_service.cc:1467] Processing CreateTablet for tablet ddc6fd5142f04f7b8feb912dbfe38b5c (DEFAULT_TABLE table=test-table [id=714860cc3f1745058a7704c36e9ea5cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.967080 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc6fd5142f04f7b8feb912dbfe38b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.967080 8222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc6fd5142f04f7b8feb912dbfe38b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.967620 8481 ts_tablet_manager.cc:1905] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:27.967868 8362 tablet_service.cc:1467] Processing CreateTablet for tablet ddc6fd5142f04f7b8feb912dbfe38b5c (DEFAULT_TABLE table=test-table [id=714860cc3f1745058a7704c36e9ea5cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:27.967988 8362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc6fd5142f04f7b8feb912dbfe38b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:27.969035 8481 ts_tablet_manager.cc:1918] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:27.969108 8481 log.cc:1196] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/dfecca8078c441fb8b1882bcb259dc2d
I20260619 20:50:27.969142 8457 tablet_bootstrap.cc:492] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Bootstrap starting.
I20260619 20:50:27.969326 8479 ts_tablet_manager.cc:1918] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:27.969352 8481 ts_tablet_manager.cc:1939] T dfecca8078c441fb8b1882bcb259dc2d P 5866b764233f42c094831eeefdceec6f: Deleting consensus metadata
I20260619 20:50:27.969383 8479 log.cc:1196] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/dfecca8078c441fb8b1882bcb259dc2d
I20260619 20:50:27.969480 8457 tablet_bootstrap.cc:654] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.969633 8479 ts_tablet_manager.cc:1939] T dfecca8078c441fb8b1882bcb259dc2d P 41aafc9f53dd48ec91204be2a663c417: Deleting consensus metadata
I20260619 20:50:27.969904 8480 ts_tablet_manager.cc:1918] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:27.969913 7932 catalog_manager.cc:4841] TS 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751): tablet dfecca8078c441fb8b1882bcb259dc2d (table test-table [id=1e2ba1ac267b46e9a014fbc303f3a740]) successfully deleted
I20260619 20:50:27.969956 8480 log.cc:1196] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/dfecca8078c441fb8b1882bcb259dc2d
I20260619 20:50:27.970042 8456 tablet_bootstrap.cc:492] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: Bootstrap starting.
I20260619 20:50:27.970172 8480 ts_tablet_manager.cc:1939] T dfecca8078c441fb8b1882bcb259dc2d P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting consensus metadata
I20260619 20:50:27.970443 8456 tablet_bootstrap.cc:654] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.970476 7930 catalog_manager.cc:4841] TS 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807): tablet dfecca8078c441fb8b1882bcb259dc2d (table test-table [id=1e2ba1ac267b46e9a014fbc303f3a740]) successfully deleted
I20260619 20:50:27.971063 8458 tablet_bootstrap.cc:492] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Bootstrap starting.
I20260619 20:50:27.971079 8457 tablet_bootstrap.cc:492] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: No bootstrap required, opened a new log
I20260619 20:50:27.971143 8457 ts_tablet_manager.cc:1397] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:50:27.971273 8457 raft_consensus.cc:348] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.971339 8458 tablet_bootstrap.cc:654] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:27.971345 8457 raft_consensus.cc:374] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.971369 8457 raft_consensus.cc:729] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5866b764233f42c094831eeefdceec6f, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.971417 8457 consensus_queue.cc:260] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.971511 8457 ts_tablet_manager.cc:1428] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.971699 8456 tablet_bootstrap.cc:492] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: No bootstrap required, opened a new log
I20260619 20:50:27.971755 8456 ts_tablet_manager.cc:1397] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:50:27.971849 7932 catalog_manager.cc:4841] TS 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811): tablet dfecca8078c441fb8b1882bcb259dc2d (table test-table [id=1e2ba1ac267b46e9a014fbc303f3a740]) successfully deleted
I20260619 20:50:27.971892 8456 raft_consensus.cc:348] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.971966 8456 raft_consensus.cc:374] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.971992 8456 raft_consensus.cc:729] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41aafc9f53dd48ec91204be2a663c417, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.972041 8456 consensus_queue.cc:260] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.972092 8458 tablet_bootstrap.cc:492] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: No bootstrap required, opened a new log
I20260619 20:50:27.972132 8458 ts_tablet_manager.cc:1397] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:27.972143 8456 ts_tablet_manager.cc:1428] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:27.972224 8458 raft_consensus.cc:348] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.972294 8458 raft_consensus.cc:374] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:27.972316 8458 raft_consensus.cc:729] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ed3cae2d3e4f368bee5e2d3fd9a6ca, State: Initialized, Role: FOLLOWER
I20260619 20:50:27.972352 8458 consensus_queue.cc:260] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:27.972434 8458 ts_tablet_manager.cc:1428] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:28.143217 8462 raft_consensus.cc:482] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:50:28.143347 8462 raft_consensus.cc:504] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:28.143739 8462 leader_election.cc:288] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807), 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:28.147033 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.144056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59239 (local address 127.0.98.2:43807)
0619 20:50:28.144151 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:28.144152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:28.144167 (+ 15us) server_negotiation.cc:407] Connection header received
0619 20:50:28.144266 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:28.144268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:28.144280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:28.144391 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:28.144589 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.145072 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.146254 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.146626 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.146715 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.146729 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:28.146924 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 20:50:28.146959 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 20:50:28.147059 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.144047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57115 (local address 127.0.98.3:36811)
0619 20:50:28.144216 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:28.144218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:28.144227 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:50:28.144290 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:28.144291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:28.144301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:28.144392 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:28.144582 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.145112 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.146262 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.146679 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.146768 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.146781 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:28.146942 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 20:50:28.146984 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":59,"threads_started":1}
I20260619 20:50:28.147095 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.143967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43807 (local address 127.0.98.1:59239)
0619 20:50:28.144106 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:28.144122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:28.144177 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:28.144399 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:28.144400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:28.144401 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:28.144509 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.144511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.145192 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.145193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.146146 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.146148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.146755 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.146756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.146822 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.146825 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:28.146909 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:50:28.146951 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":86,"thread_start_us":70,"threads_started":1}
I20260619 20:50:28.147106 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.143975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36811 (local address 127.0.98.1:57115)
0619 20:50:28.144124 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:28.144138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:28.144215 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:28.144395 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:28.144395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:28.144397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:28.144495 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.144497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.145220 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.145221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.146156 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.146158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.146767 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.146768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.146851 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.146855 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:28.146923 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:50:28.146966 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"spinlock_wait_cycles":1408,"thread_start_us":66,"threads_started":1}
I20260619 20:50:28.147380 8242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc6fd5142f04f7b8feb912dbfe38b5c" candidate_uuid: "41aafc9f53dd48ec91204be2a663c417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" is_pre_election: true
I20260619 20:50:28.147471 8242 raft_consensus.cc:2454] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41aafc9f53dd48ec91204be2a663c417 in term 0.
I20260619 20:50:28.147428 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc6fd5142f04f7b8feb912dbfe38b5c" candidate_uuid: "41aafc9f53dd48ec91204be2a663c417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f" is_pre_election: true
I20260619 20:50:28.147533 8382 raft_consensus.cc:2454] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41aafc9f53dd48ec91204be2a663c417 in term 0.
I20260619 20:50:28.147689 8034 leader_election.cc:302] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aafc9f53dd48ec91204be2a663c417, 52ed3cae2d3e4f368bee5e2d3fd9a6ca; no voters:
I20260619 20:50:28.147845 8462 raft_consensus.cc:2781] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:50:28.147919 8462 raft_consensus.cc:482] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:50:28.147938 8462 raft_consensus.cc:3037] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:28.148636 8462 raft_consensus.cc:504] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:28.148808 8462 leader_election.cc:288] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [CANDIDATE]: Term 1 election: Requested vote from peers 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807), 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811)
I20260619 20:50:28.148963 8242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc6fd5142f04f7b8feb912dbfe38b5c" candidate_uuid: "41aafc9f53dd48ec91204be2a663c417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca"
I20260619 20:50:28.149019 8382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc6fd5142f04f7b8feb912dbfe38b5c" candidate_uuid: "41aafc9f53dd48ec91204be2a663c417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5866b764233f42c094831eeefdceec6f"
I20260619 20:50:28.149065 8242 raft_consensus.cc:3037] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:28.149080 8382 raft_consensus.cc:3037] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:28.149883 8382 raft_consensus.cc:2454] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41aafc9f53dd48ec91204be2a663c417 in term 1.
I20260619 20:50:28.149883 8242 raft_consensus.cc:2454] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41aafc9f53dd48ec91204be2a663c417 in term 1.
I20260619 20:50:28.150065 8034 leader_election.cc:302] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41aafc9f53dd48ec91204be2a663c417, 52ed3cae2d3e4f368bee5e2d3fd9a6ca; no voters:
I20260619 20:50:28.150174 8462 raft_consensus.cc:2781] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:28.150303 8462 raft_consensus.cc:686] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 1 LEADER]: Becoming Leader. State: Replica: 41aafc9f53dd48ec91204be2a663c417, State: Running, Role: LEADER
I20260619 20:50:28.150416 8462 consensus_queue.cc:237] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } }
I20260619 20:50:28.151119 7945 catalog_manager.cc:5495] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "41aafc9f53dd48ec91204be2a663c417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41aafc9f53dd48ec91204be2a663c417" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 } health_report { overall_health: UNKNOWN } } }
I20260619 20:50:28.199864 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.197584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.98.1:44751)
0619 20:50:28.197759 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:28.197760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:28.197767 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:50:28.197800 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:28.197800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:28.197818 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:28.197889 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:28.197981 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.198460 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.199130 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.199320 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.199369 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.199488 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:28.199620 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:28.199713 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:28.199760 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 20:50:28.199778 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260619 20:50:28.204327 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.202282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60533 (local address 127.0.98.2:43807)
0619 20:50:28.202337 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:28.202339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:28.202347 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:28.202387 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:28.202388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:28.202396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:28.202449 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:28.202560 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.202997 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.203690 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.203858 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.203904 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.204068 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:28.204103 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:28.204168 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:28.204238 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:50:28.204258 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:50:28.204352 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.202129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43807 (local address 127.0.98.1:60533)
0619 20:50:28.202287 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:28.202298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:28.202347 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:28.202454 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:28.202455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:28.202456 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:50:28.202504 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.202505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.203057 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.203058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.203627 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.203629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.203920 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.203920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.203963 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.203973 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:50:28.204184 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:50:28.204186 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:28.204234 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:50:28.204270 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260619 20:50:28.206408 8242 raft_consensus.cc:1261] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Refusing update from remote peer 41aafc9f53dd48ec91204be2a663c417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:28.206453 8382 raft_consensus.cc:1261] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Refusing update from remote peer 41aafc9f53dd48ec91204be2a663c417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:28.206642 8537 consensus_queue.cc:1035] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ed3cae2d3e4f368bee5e2d3fd9a6ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:28.206764 8462 consensus_queue.cc:1035] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5866b764233f42c094831eeefdceec6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:28.213131 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.210974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36425 (local address 127.0.98.1:44751)
0619 20:50:28.211024 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:28.211026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:28.211102 (+ 76us) server_negotiation.cc:407] Connection header received
0619 20:50:28.211170 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:28.211170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:28.211178 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:28.211217 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:28.211339 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.211795 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.212478 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.212644 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.212691 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.212879 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:28.212913 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:28.212975 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:28.213053 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:50:28.213071 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:50:28.213168 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.210915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44751 (local address 127.0.98.2:36425)
0619 20:50:28.211042 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:28.211053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:28.211113 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:28.211227 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:28.211228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:28.211230 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:50:28.211269 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.211270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.211876 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.211877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.212410 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:28.212411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:28.212718 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:28.212718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:28.212754 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:28.212758 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:50:28.212975 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:50:28.212977 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:28.213046 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:50:28.213092 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260619 20:50:28.219820 7944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:28.219919 7944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:28.220539 7944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b50142f8de5406bba0bc96e2f9af314: Sending DeleteTablet for 3 replicas of tablet ddc6fd5142f04f7b8feb912dbfe38b5c
I20260619 20:50:28.220786 8222 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc6fd5142f04f7b8feb912dbfe38b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260619 20:50:28.220803 8362 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc6fd5142f04f7b8feb912dbfe38b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51986
I20260619 20:50:28.220814 8075 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc6fd5142f04f7b8feb912dbfe38b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39080
I20260619 20:50:28.220870 8480 tablet_replica.cc:330] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: stopping tablet replica
I20260619 20:50:28.220887 8479 tablet_replica.cc:330] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: stopping tablet replica
I20260619 20:50:28.220887 8481 tablet_replica.cc:330] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: stopping tablet replica
I20260619 20:50:28.220916 8480 raft_consensus.cc:2229] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:28.220924 8479 raft_consensus.cc:2229] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:50:28.220932 8481 raft_consensus.cc:2229] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:28.220952 8480 raft_consensus.cc:2258] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:28.220965 8481 raft_consensus.cc:2258] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:28.220993 8479 raft_consensus.cc:2258] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:28.221355 8479 ts_tablet_manager.cc:1905] T ddc6fd5142f04f7b8feb912dbfe38b5c P 41aafc9f53dd48ec91204be2a663c417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:28.221391 8480 ts_tablet_manager.cc:1905] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:28.221500 8481 ts_tablet_manager.cc:1905] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:28.222998 8480 ts_tablet_manager.cc:1918] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:50:28.222999 8481 ts_tablet_manager.cc:1918] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:50:28.223060 8480 log.cc:1196] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/ddc6fd5142f04f7b8feb912dbfe38b5c
I20260619 20:50:28.223060 8481 log.cc:1196] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/ddc6fd5142f04f7b8feb912dbfe38b5c
I20260619 20:50:28.223165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8017
I20260619 20:50:28.223304 8481 ts_tablet_manager.cc:1939] T ddc6fd5142f04f7b8feb912dbfe38b5c P 5866b764233f42c094831eeefdceec6f: Deleting consensus metadata
I20260619 20:50:28.223325 8480 ts_tablet_manager.cc:1939] T ddc6fd5142f04f7b8feb912dbfe38b5c P 52ed3cae2d3e4f368bee5e2d3fd9a6ca: Deleting consensus metadata
I20260619 20:50:28.223845 7930 catalog_manager.cc:4841] TS 52ed3cae2d3e4f368bee5e2d3fd9a6ca (127.0.98.2:43807): tablet ddc6fd5142f04f7b8feb912dbfe38b5c (table test-table [id=714860cc3f1745058a7704c36e9ea5cc]) successfully deleted
I20260619 20:50:28.224526 7932 catalog_manager.cc:4841] TS 5866b764233f42c094831eeefdceec6f (127.0.98.3:36811): tablet ddc6fd5142f04f7b8feb912dbfe38b5c (table test-table [id=714860cc3f1745058a7704c36e9ea5cc]) successfully deleted
W20260619 20:50:28.228897 7932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44751 (error 108)
I20260619 20:50:28.229132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8157
I20260619 20:50:28.229429 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:28.229248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44751 (local address 127.0.0.1:39086)
0619 20:50:28.229349 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:28.229386 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":36,"threads_started":1}
W20260619 20:50:28.229545 7932 catalog_manager.cc:4584] TS 41aafc9f53dd48ec91204be2a663c417 (127.0.98.1:44751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddc6fd5142f04f7b8feb912dbfe38b5c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44751: connect: Connection refused (error 111)
I20260619 20:50:28.236038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8297
I20260619 20:50:28.241328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 7913
2026-06-19T20:50:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3161 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:50:28 dist-test-slave-0smx krb5kdc[8554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:50:28 dist-test-slave-0smx krb5kdc[8554](info): set up 2 sockets
Jun 19 20:50:28 dist-test-slave-0smx krb5kdc[8554](info): commencing operation
krb5kdc: starting...
W20260619 20:50:29.277774 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:50:29 dist-test-slave-0smx krb5kdc[8554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:50:29.388649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:29.466753 8567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:29.466985 8567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:29.467021 8567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:29.468431 8567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:29.468515 8567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:29.468544 8567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:29.468571 8567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:29.468626 8567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:29.470363 8567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:29.470645 8567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:29.470873 8567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:29.475808 8573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:29.475826 8572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:29.475826 8575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:29.476089 8567 server_base.cc:1034] running on GCE node
I20260619 20:50:29.476284 8567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:29.476541 8567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:29.477696 8567 hybrid_clock.cc:648] HybridClock initialized: now 1781902229477683 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T20:50:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1243 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:50:29 dist-test-slave-0smx krb5kdc[8580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:50:29 dist-test-slave-0smx krb5kdc[8580](info): set up 2 sockets
Jun 19 20:50:29 dist-test-slave-0smx krb5kdc[8580](info): commencing operation
krb5kdc: starting...
W20260619 20:50:30.518121 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:50:30 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:50:30.630128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:30.708335 8593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:30.708564 8593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:30.708601 8593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:30.709860 8593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:30.709926 8593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:30.709949 8593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:30.709971 8593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:30.710000 8593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:30.711452 8593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:30.711710 8593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:30.711946 8593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:30.716595 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:30.716601 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:30.716607 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:30.716822 8593 server_base.cc:1034] running on GCE node
I20260619 20:50:30.717056 8593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:30.717417 8593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:30.718582 8593 hybrid_clock.cc:648] HybridClock initialized: now 1781902230718569 us; error 44 us; skew 500 ppm
Jun 19 20:50:30 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:30.724306 8593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:50:30.724740 8593 webserver.cc:460] Webserver started at http://127.0.98.62:46523/ using document root <none> and password file <none>
I20260619 20:50:30.724957 8593 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:30.724998 8593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:30.725083 8593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:30.726107 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "f1441e21f2af4c099f236241a87189f0"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "4e03b062ff2e57245c16ad411d962a74"
server_key_iv: "d3d3bb2f98b71aaf083d870194c1e312"
server_key_version: "encryptionkey@0"
I20260619 20:50:30.726388 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1441e21f2af4c099f236241a87189f0"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "4e03b062ff2e57245c16ad411d962a74"
server_key_iv: "d3d3bb2f98b71aaf083d870194c1e312"
server_key_version: "encryptionkey@0"
I20260619 20:50:30.727603 8593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:50:30.728273 8608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:30.728439 8593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:50:30.728519 8593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "f1441e21f2af4c099f236241a87189f0"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "4e03b062ff2e57245c16ad411d962a74"
server_key_iv: "d3d3bb2f98b71aaf083d870194c1e312"
server_key_version: "encryptionkey@0"
I20260619 20:50:30.728572 8593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:30.739457 8593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:30.741170 8593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:30.741273 8593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:30.744861 8593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44327
I20260619 20:50:30.744910 8660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44327 every 8 connection(s)
I20260619 20:50:30.745452 8593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:50:30.746109 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:30.748201 8661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Bootstrap starting.
I20260619 20:50:30.748798 8661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:30.749102 8661 log.cc:824] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:30.749765 8661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: No bootstrap required, opened a new log
I20260619 20:50:30.751238 8661 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:30.751396 8661 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:30.751442 8661 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1441e21f2af4c099f236241a87189f0, State: Initialized, Role: FOLLOWER
I20260619 20:50:30.751617 8661 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:30.751714 8661 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:50:30.751760 8661 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:50:30.751808 8661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:30.752705 8661 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:30.752818 8661 leader_election.cc:302] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1441e21f2af4c099f236241a87189f0; no voters:
I20260619 20:50:30.752993 8661 leader_election.cc:288] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:50:30.753027 8664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:30.753250 8664 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 LEADER]: Becoming Leader. State: Replica: f1441e21f2af4c099f236241a87189f0, State: Running, Role: LEADER
I20260619 20:50:30.753249 8661 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:50:30.753437 8664 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:30.753772 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1441e21f2af4c099f236241a87189f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } } }
I20260619 20:50:30.753782 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1441e21f2af4c099f236241a87189f0. Latest consensus state: current_term: 1 leader_uuid: "f1441e21f2af4c099f236241a87189f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } } }
I20260619 20:50:30.753892 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:30.753975 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:30.754181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8593
I20260619 20:50:30.754206 8673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:50:30.754271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:50:30.754485 392 external_mini_cluster.cc:1428] Setting key 64299a48d5047d0e763c876b37bc005e
I20260619 20:50:30.754704 8673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:50:30.756773 8673 catalog_manager.cc:1347] Generated new cluster ID: 75085b0499b74f1b947ab2216535111d
I20260619 20:50:30.756829 8673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:50:30 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:30.763948 8673 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:50:30.764616 8673 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:50:30.764823 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:30.755157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.0.98.62:44327)
0619 20:50:30.755329 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:30.755338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:30.755367 (+ 29us) server_negotiation.cc:407] Connection header received
0619 20:50:30.755494 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:30.755501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:30.755557 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:30.755712 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:30.756473 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:30.757635 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:30.758536 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:30.758979 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:30.759095 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:30.761474 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:30.761478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:30.761478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:30.761490 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:30.763138 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:30.763499 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:30.763502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:30.763502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:30.763639 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:30.763874 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:30.763875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:30.763875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:30.764142 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:30.764322 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:30.764393 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 20:50:30.764529 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260619 20:50:30.770934 8673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Generated new TSK 0
I20260619 20:50:30.771107 8673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:50:30.800366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44327
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:30.899655 8687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:30.899880 8687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:30.899916 8687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:30.901021 8687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:30.901084 8687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:30.901127 8687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:50:30.902617 8687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:30.902897 8687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:30.903113 8687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:30.908928 8693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:30.909032 8695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:30.908939 8692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:30.909212 8687 server_base.cc:1034] running on GCE node
I20260619 20:50:30.909502 8687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:30.909739 8687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:30.910897 8687 hybrid_clock.cc:648] HybridClock initialized: now 1781902230910880 us; error 27 us; skew 500 ppm
Jun 19 20:50:30 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:30.913682 8687 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:50:30.914161 8687 webserver.cc:460] Webserver started at http://127.0.98.1:36945/ using document root <none> and password file <none>
I20260619 20:50:30.914382 8687 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:30.914448 8687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:30.914588 8687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:30.915632 8687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "dfd1c1289a83414fbf4b18f27158f47d"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "c8f49ab40471ea2b0c9a3d371fb3c753"
server_key_iv: "c6f301405c15e5f4d691cbaa829c76f2"
server_key_version: "encryptionkey@0"
I20260619 20:50:30.915987 8687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dfd1c1289a83414fbf4b18f27158f47d"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "c8f49ab40471ea2b0c9a3d371fb3c753"
server_key_iv: "c6f301405c15e5f4d691cbaa829c76f2"
server_key_version: "encryptionkey@0"
I20260619 20:50:30.917364 8687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:50:30.918134 8702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:30.918318 8687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:30.918413 8687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "dfd1c1289a83414fbf4b18f27158f47d"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "c8f49ab40471ea2b0c9a3d371fb3c753"
server_key_iv: "c6f301405c15e5f4d691cbaa829c76f2"
server_key_version: "encryptionkey@0"
I20260619 20:50:30.918494 8687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:30.934830 8687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:30.936685 8687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:30.936843 8687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:30.937063 8687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:30.937399 8687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:30.937453 8687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:30.937486 8687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:30.937512 8687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:30.943292 8687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32811
I20260619 20:50:30.943331 8815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32811 every 8 connection(s)
I20260619 20:50:30.943631 8687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:50:30.945510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8687
I20260619 20:50:30.945622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:50:30.945844 392 external_mini_cluster.cc:1428] Setting key e2deb09e2e5bc00126b0171d3599ed79
Jun 19 20:50:30 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:30.953943 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:30.944285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40521 (local address 127.0.98.62:44327)
0619 20:50:30.944370 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:30.944372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:30.944555 (+ 183us) server_negotiation.cc:407] Connection header received
0619 20:50:30.945538 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:30.945540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:30.945553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:30.945646 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:30.946381 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:30.947101 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:30.948365 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:30.948558 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:30.948614 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:30.951047 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:30.951048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:30.951048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:30.951055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:30.952561 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:30.952991 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:30.952992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:30.952993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:30.953054 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:30.953320 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:30.953321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:30.953321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:30.953441 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:30.953516 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:30.953780 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 20:50:30.953854 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:50:30.954131 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:30.944265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:40521)
0619 20:50:30.944435 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:30.944469 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:30.945421 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:30.945744 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:30.945748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:30.946068 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:30.946299 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:30.946304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:30.947194 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:30.947195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:30.948266 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:30.948269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:30.948639 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:30.948640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:30.948785 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:30.949264 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:30.949273 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:30.950936 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:30.952670 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:30.952677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:30.952679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:30.952913 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:30.953121 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:30.953122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:30.953122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:30.953229 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:30.953508 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:30.953514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:30.953647 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:50:30.953744 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 20:50:30.953902 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260619 20:50:30.954640 8816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44327
I20260619 20:50:30.954752 8816 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:30.954929 8816 heartbeater.cc:511] Master 127.0.98.62:44327 requested a full tablet report, sending...
I20260619 20:50:30.955504 8625 ts_manager.cc:194] Registered new tserver with Master: dfd1c1289a83414fbf4b18f27158f47d (127.0.98.1:32811)
I20260619 20:50:30.956169 8625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:50:30.970451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44327
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:31.048702 8821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:31.048902 8821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:31.048925 8821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:31.050122 8821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:31.050163 8821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:31.050189 8821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:50:31.051643 8821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:31.051889 8821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:31.052109 8821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:31.057174 8827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.057184 8821 server_base.cc:1034] running on GCE node
W20260619 20:50:31.057168 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.057169 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.057627 8821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:31.057868 8821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:31.059002 8821 hybrid_clock.cc:648] HybridClock initialized: now 1781902231058992 us; error 27 us; skew 500 ppm
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:31.061463 8821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:50:31.061867 8821 webserver.cc:460] Webserver started at http://127.0.98.2:42351/ using document root <none> and password file <none>
I20260619 20:50:31.062083 8821 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:31.062127 8821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:31.062268 8821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:31.063234 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "863fb0cd30da42c6be0a379b8fe4142c"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "40f4b3ac4eab88ca9d85f02b22a46f13"
server_key_iv: "39f262083b1ecc82cfe36fafc91a4448"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.063573 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "863fb0cd30da42c6be0a379b8fe4142c"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "40f4b3ac4eab88ca9d85f02b22a46f13"
server_key_iv: "39f262083b1ecc82cfe36fafc91a4448"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.064846 8821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:50:31.065567 8836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.065747 8821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:31.065814 8821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "863fb0cd30da42c6be0a379b8fe4142c"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "40f4b3ac4eab88ca9d85f02b22a46f13"
server_key_iv: "39f262083b1ecc82cfe36fafc91a4448"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.065897 8821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:31.074437 8821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:31.076155 8821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:31.076301 8821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:31.076508 8821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:31.076800 8821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:31.076856 8821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.076894 8821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:31.076922 8821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.082617 8821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39291
I20260619 20:50:31.082670 8949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39291 every 8 connection(s)
I20260619 20:50:31.082973 8821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:50:31.084625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8821
I20260619 20:50:31.084753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:50:31.084968 392 external_mini_cluster.cc:1428] Setting key 6ade99866481a2e0b7afda01088e4539
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:31.093039 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.083630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37081 (local address 127.0.98.62:44327)
0619 20:50:31.083688 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.083690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.083872 (+ 182us) server_negotiation.cc:407] Connection header received
0619 20:50:31.084971 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.084972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.084981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.085071 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.085756 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.086220 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.087507 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.087821 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.087896 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.090276 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:31.090277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:31.090277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:31.090283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:31.091650 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:31.092132 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:31.092133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:31.092133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:31.092182 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:31.092444 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:31.092444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:31.092444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:31.092536 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:31.092584 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:31.092913 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 20:50:31.092975 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:50:31.093353 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.083601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:37081)
0619 20:50:31.083745 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.083785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.084868 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.085159 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.085163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.085460 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.085645 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.085649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.086313 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.086314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.087421 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.087424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.087904 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.087904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.087998 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.088492 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.088500 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.090186 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:31.091735 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:31.091741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:31.091743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:31.092058 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:31.092244 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:31.092245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:31.092245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:31.092380 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:31.092600 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:31.092606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:31.092771 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 20:50:31.092897 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 20:50:31.093069 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:50:31.093845 8950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44327
I20260619 20:50:31.093935 8950 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:31.094111 8950 heartbeater.cc:511] Master 127.0.98.62:44327 requested a full tablet report, sending...
I20260619 20:50:31.094453 8625 ts_manager.cc:194] Registered new tserver with Master: 863fb0cd30da42c6be0a379b8fe4142c (127.0.98.2:39291)
I20260619 20:50:31.094836 8625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:50:31.109270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44327
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:31.190419 8955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:31.190657 8955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:31.190706 8955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:31.191869 8955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:31.191947 8955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:31.191989 8955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:50:31.193519 8955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:31.193780 8955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:31.194000 8955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:31.199456 8961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.199461 8963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.199530 8955 server_base.cc:1034] running on GCE node
W20260619 20:50:31.199476 8960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.199908 8955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:31.200162 8955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:31.201342 8955 hybrid_clock.cc:648] HybridClock initialized: now 1781902231201330 us; error 27 us; skew 500 ppm
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:31.203835 8955 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:50:31.204267 8955 webserver.cc:460] Webserver started at http://127.0.98.3:41043/ using document root <none> and password file <none>
I20260619 20:50:31.204509 8955 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:31.204576 8955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:31.204725 8955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:31.205821 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "741fb200dbe3491ba9eec7858cefaa82"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "15daa558b8281a74a87b6e34b6a7246d"
server_key_iv: "55b43ebf55e0b32adf9251bbb2000b95"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.206176 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "741fb200dbe3491ba9eec7858cefaa82"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "15daa558b8281a74a87b6e34b6a7246d"
server_key_iv: "55b43ebf55e0b32adf9251bbb2000b95"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.207472 8955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:31.208240 8970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.208406 8955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:31.208501 8955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "741fb200dbe3491ba9eec7858cefaa82"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "15daa558b8281a74a87b6e34b6a7246d"
server_key_iv: "55b43ebf55e0b32adf9251bbb2000b95"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.208576 8955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:31.240118 8955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:31.242136 8955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:31.242301 8955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:31.242542 8955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:31.242882 8955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:31.242938 8955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.242983 8955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:31.243012 8955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.248766 8955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36591
I20260619 20:50:31.248808 9083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36591 every 8 connection(s)
I20260619 20:50:31.249164 8955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:50:31.253712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 8955
I20260619 20:50:31.253852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:50:31.254041 392 external_mini_cluster.cc:1428] Setting key 3ff08f729202305e8251441e9c8d0e47
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:31.259136 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.249978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48903 (local address 127.0.98.62:44327)
0619 20:50:31.250021 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.250023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.250239 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:50:31.251215 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.251217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.251229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.251294 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.251944 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.252454 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.253721 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.254076 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.254150 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.256473 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:31.256475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:31.256475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:31.256481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:31.257910 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:31.258246 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:31.258246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:31.258247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:31.258297 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:31.258499 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:31.258499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:31.258499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:31.258602 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:31.258699 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:31.259013 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 20:50:31.259074 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:50:31.259348 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.249944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:48903)
0619 20:50:31.250120 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.250157 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.251100 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.251377 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.251381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.251646 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.251843 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.251848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.252553 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.252554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.253628 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.253631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.254158 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.254159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.254361 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.254998 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.255006 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.256391 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:31.257975 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:31.257979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:31.257981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:31.258183 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:31.258353 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:31.258353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:31.258353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:31.258446 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:31.258761 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:31.258766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:31.258893 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 20:50:31.258980 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:50:31.259097 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260619 20:50:31.259807 9084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44327
I20260619 20:50:31.259917 9084 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:31.260107 9084 heartbeater.cc:511] Master 127.0.98.62:44327 requested a full tablet report, sending...
I20260619 20:50:31.260493 8625 ts_manager.cc:194] Registered new tserver with Master: 741fb200dbe3491ba9eec7858cefaa82 (127.0.98.3:36591)
I20260619 20:50:31.260849 8625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48903
I20260619 20:50:31.265859 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 20:50:31.331470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8687
I20260619 20:50:31.335978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8821
I20260619 20:50:31.340114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8955
I20260619 20:50:31.344442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 8593
I20260619 20:50:31.349565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44327
--webserver_interface=127.0.98.62
--webserver_port=46523
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:31.428035 9093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:31.428269 9093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:31.428306 9093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:31.429451 9093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:31.429517 9093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:31.429540 9093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:31.429561 9093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:31.429592 9093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:31.431056 9093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:31.431326 9093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:31.431548 9093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:31.436381 9099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.436383 9098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.436383 9101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.436707 9093 server_base.cc:1034] running on GCE node
I20260619 20:50:31.436941 9093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:31.437217 9093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:31.438439 9093 hybrid_clock.cc:648] HybridClock initialized: now 1781902231438444 us; error 50 us; skew 500 ppm
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:31.440876 9093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:50:31.441260 9093 webserver.cc:460] Webserver started at http://127.0.98.62:46523/ using document root <none> and password file <none>
I20260619 20:50:31.441519 9093 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:31.441576 9093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:31.443099 9093 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:50:31.443815 9108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.443981 9093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:50:31.444061 9093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "f1441e21f2af4c099f236241a87189f0"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "4e03b062ff2e57245c16ad411d962a74"
server_key_iv: "d3d3bb2f98b71aaf083d870194c1e312"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.444388 9093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:31.458902 9093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:31.460906 9093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:31.461011 9093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:31.464772 9093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44327
I20260619 20:50:31.464818 9160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44327 every 8 connection(s)
I20260619 20:50:31.465334 9093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:50:31.466548 9161 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:50:31.467103 9161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Bootstrap starting.
I20260619 20:50:31.474289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9093
I20260619 20:50:31.474627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32811
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44327
--webserver_port=36945
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:50:31.475145 9161 log.cc:824] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:31.476887 9161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:50:31.477160 9161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Bootstrap complete.
I20260619 20:50:31.479429 9161 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:31.479641 9161 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1441e21f2af4c099f236241a87189f0, State: Initialized, Role: FOLLOWER
I20260619 20:50:31.479799 9161 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:31.479874 9161 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:50:31.479935 9161 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:50:31.479975 9161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:50:31.480942 9161 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:31.481025 9161 leader_election.cc:302] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1441e21f2af4c099f236241a87189f0; no voters:
I20260619 20:50:31.481240 9161 leader_election.cc:288] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:50:31.481267 9165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:50:31.481393 9165 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [term 2 LEADER]: Becoming Leader. State: Replica: f1441e21f2af4c099f236241a87189f0, State: Running, Role: LEADER
I20260619 20:50:31.481483 9165 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } }
I20260619 20:50:31.481520 9161 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:50:31.481745 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1441e21f2af4c099f236241a87189f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } } }
I20260619 20:50:31.481812 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:31.481922 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1441e21f2af4c099f236241a87189f0. Latest consensus state: current_term: 2 leader_uuid: "f1441e21f2af4c099f236241a87189f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1441e21f2af4c099f236241a87189f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44327 } } }
I20260619 20:50:31.481992 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:31.482467 9169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:50:31.482975 9169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:50:31.483116 9169 catalog_manager.cc:1259] Loaded cluster ID: 75085b0499b74f1b947ab2216535111d
I20260619 20:50:31.483304 9169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:50:31.484094 9169 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:50:31.484283 9169 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1441e21f2af4c099f236241a87189f0: Loaded TSK: 0
I20260619 20:50:31.484510 9169 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:50:31.569195 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:31.569454 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:31.569507 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:31.570787 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:31.570845 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:31.570888 9164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:50:31.572284 9164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36945
--tserver_master_addrs=127.0.98.62:44327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:31.572552 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:31.572785 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:31.578802 9187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.579559 9164 server_base.cc:1034] running on GCE node
W20260619 20:50:31.579769 9190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.580106 9188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.580372 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:31.580657 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:31.581786 9164 hybrid_clock.cc:648] HybridClock initialized: now 1781902231581769 us; error 34 us; skew 500 ppm
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:31.585636 9164 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 20:50:31.586175 9164 webserver.cc:460] Webserver started at http://127.0.98.1:36945/ using document root <none> and password file <none>
I20260619 20:50:31.586440 9164 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:31.586525 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:31.589054 9164 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 20:50:31.589761 9197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.589921 9164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:50:31.590030 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "dfd1c1289a83414fbf4b18f27158f47d"
format_stamp: "Formatted at 2026-06-19 20:50:30 on dist-test-slave-0smx"
server_key: "c8f49ab40471ea2b0c9a3d371fb3c753"
server_key_iv: "c6f301405c15e5f4d691cbaa829c76f2"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.590402 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:31.607769 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:31.609941 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:31.610100 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:31.610338 9164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:31.610632 9164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:31.610687 9164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.610756 9164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:31.610787 9164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.616768 9164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32811
I20260619 20:50:31.616848 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32811 every 8 connection(s)
I20260619 20:50:31.617139 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:50:31.619896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9164
I20260619 20:50:31.620218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39291
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44327
--webserver_port=42351
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.630839 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.617893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:46021)
0619 20:50:31.618076 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.618120 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.618566 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.621923 (+ 3357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.621929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.622670 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.622939 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.622945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.624726 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.624727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.626067 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.626070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.626988 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.626989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.627132 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.627712 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.627723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.630624 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
W20260619 20:50:31.631227 9311 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44327 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44327: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260619 20:50:31.631671 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.617953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46021 (local address 127.0.98.62:44327)
0619 20:50:31.621425 (+ 3472us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.621436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.621479 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:50:31.621629 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.621636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.621693 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.621852 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.623074 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.624614 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.626182 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.626885 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.627002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.630779 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46021: BlockingRecv error: recv got EOF from 127.0.98.1:46021 (error 108)
Metrics: {"server-negotiator.queue_time_us":3417,"thread_start_us":101,"threads_started":1}
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.637140 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.631452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:58677)
0619 20:50:31.631495 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.631504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.631536 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.631945 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.631946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.632093 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.632280 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.632282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.633219 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.633220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.634104 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.634106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.634567 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.634568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.634649 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.635097 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.635102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.637068 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:50:31.637265 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.631509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58677 (local address 127.0.98.62:44327)
0619 20:50:31.631814 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.631816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.631824 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:31.631845 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.631845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.631857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.631906 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.632377 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.633111 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.634207 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.634465 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.634543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.637191 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58677: BlockingRecv error: recv got EOF from 127.0.98.1:58677 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":173,"threads_started":1}
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.641957 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.637387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:55483)
0619 20:50:31.637512 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.637523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.637585 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.637731 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.637732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.637880 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.637918 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.637920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.638490 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.638491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.639296 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.639298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.639602 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.639603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.639655 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.640001 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.640006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.641885 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260619 20:50:31.642071 9311 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:50:31.642089 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.637422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55483 (local address 127.0.98.62:44327)
0619 20:50:31.637462 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.637463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.637584 (+ 121us) server_negotiation.cc:407] Connection header received
0619 20:50:31.637645 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.637645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.637655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.637704 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.637990 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.638420 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.639371 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.639533 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.639579 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.641987 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55483: BlockingRecv error: recv got EOF from 127.0.98.1:55483 (error 108)
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":25}
W20260619 20:50:31.706604 9315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:31.706807 9315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:31.706831 9315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:31.707958 9315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:31.707994 9315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:31.708019 9315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:50:31.709381 9315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42351
--tserver_master_addrs=127.0.98.62:44327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:31.709553 9315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:31.709719 9315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:31.714664 9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.714645 9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.714689 9315 server_base.cc:1034] running on GCE node
W20260619 20:50:31.714633 9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.715171 9315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:31.715425 9315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:31.716607 9315 hybrid_clock.cc:648] HybridClock initialized: now 1781902231716591 us; error 29 us; skew 500 ppm
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:31.719045 9315 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 20:50:31.719519 9315 webserver.cc:460] Webserver started at http://127.0.98.2:42351/ using document root <none> and password file <none>
I20260619 20:50:31.719741 9315 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:31.719823 9315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:31.721182 9315 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:31.721832 9332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.721992 9315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.722087 9315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "863fb0cd30da42c6be0a379b8fe4142c"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "40f4b3ac4eab88ca9d85f02b22a46f13"
server_key_iv: "39f262083b1ecc82cfe36fafc91a4448"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.722440 9315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:31.738181 9315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:31.740371 9315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:31.740554 9315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:31.740819 9315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:31.741127 9315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:31.741183 9315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.741227 9315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:31.741261 9315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.747141 9315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39291
I20260619 20:50:31.747184 9445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39291 every 8 connection(s)
I20260619 20:50:31.747560 9315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:50:31.754508 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.748305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57005 (local address 127.0.98.62:44327)
0619 20:50:31.748354 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.748357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.748599 (+ 242us) server_negotiation.cc:407] Connection header received
0619 20:50:31.749063 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.749064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.749077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.749131 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.750095 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.750566 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.751542 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.751730 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.751793 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.754465 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57005: BlockingRecv error: recv got EOF from 127.0.98.2:57005 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:50:31.754516 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.748301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:57005)
0619 20:50:31.748469 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.748503 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.748983 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.749209 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.749213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.749816 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.750003 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.750007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.750648 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.750649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.751460 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.751462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.751811 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.751812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.751902 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.752316 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.752323 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.754389 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
W20260619 20:50:31.754716 9446 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44327 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44327: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 20:50:31.755353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9315
I20260619 20:50:31.755623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36591
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44327
--webserver_port=41043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.759388 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.754957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:54423)
0619 20:50:31.755066 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.755087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.755156 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.755296 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.755297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.755464 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.755513 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.755518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.756209 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.756210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.756698 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.756700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.757043 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.757044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.757112 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.757559 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.757563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.759297 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260619 20:50:31.759408 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.755027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54423 (local address 127.0.98.62:44327)
0619 20:50:31.755062 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.755064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.755143 (+ 79us) server_negotiation.cc:407] Connection header received
0619 20:50:31.755215 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.755216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.755224 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.755292 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.755593 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.756135 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.756778 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.756967 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.757044 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.759369 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54423: BlockingRecv error: recv got EOF from 127.0.98.2:54423 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.764441 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.759870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:48981)
0619 20:50:31.759978 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.759989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.760034 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.760211 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.760212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.760371 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.760406 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.760408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.760987 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.760988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.761901 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.761904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.762232 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.762233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.762293 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.762632 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.762638 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.764365 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260619 20:50:31.764485 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.759933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48981 (local address 127.0.98.62:44327)
0619 20:50:31.759971 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.759973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.760021 (+ 48us) server_negotiation.cc:407] Connection header received
0619 20:50:31.760099 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.760099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.760106 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.760181 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.760478 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.760914 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.762000 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.762163 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.762211 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.764426 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48981: BlockingRecv error: recv got EOF from 127.0.98.2:48981 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:50:31.764576 9446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:50:31.835218 9449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:31.835443 9449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:31.835477 9449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:31.836637 9449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:31.836704 9449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:31.836745 9449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:50:31.838158 9449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41043
--tserver_master_addrs=127.0.98.62:44327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:31.838416 9449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:31.838642 9449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:31.843585 9456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:31.843595 9455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.843611 9449 server_base.cc:1034] running on GCE node
W20260619 20:50:31.843595 9458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:31.844036 9449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:31.844285 9449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:31.845381 9449 hybrid_clock.cc:648] HybridClock initialized: now 1781902231845370 us; error 26 us; skew 500 ppm
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902231, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:31.847719 9449 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 20:50:31.848225 9449 webserver.cc:460] Webserver started at http://127.0.98.3:41043/ using document root <none> and password file <none>
I20260619 20:50:31.848452 9449 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:31.848532 9449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:31.849900 9449 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:31.850479 9465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.850641 9449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:31.850745 9449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "741fb200dbe3491ba9eec7858cefaa82"
format_stamp: "Formatted at 2026-06-19 20:50:31 on dist-test-slave-0smx"
server_key: "15daa558b8281a74a87b6e34b6a7246d"
server_key_iv: "55b43ebf55e0b32adf9251bbb2000b95"
server_key_version: "encryptionkey@0"
I20260619 20:50:31.851075 9449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:31.862103 9449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:31.863819 9449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:31.863977 9449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:31.864202 9449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:31.864493 9449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:31.864545 9449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.864586 9449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:31.864614 9449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:31.870352 9449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36591
I20260619 20:50:31.870401 9578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36591 every 8 connection(s)
I20260619 20:50:31.870739 9449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:50:31.877686 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.871413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33943 (local address 127.0.98.62:44327)
0619 20:50:31.871485 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.871487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.871678 (+ 191us) server_negotiation.cc:407] Connection header received
0619 20:50:31.872157 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.872158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.872168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.872227 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.873053 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.873524 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.874511 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.874680 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.874727 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.877638 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33943: BlockingRecv error: recv got EOF from 127.0.98.3:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:50:31.877697 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.871397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:33943)
0619 20:50:31.871550 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.871588 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.872056 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.872279 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.872282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.872811 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.872984 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.872988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.873593 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.873594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.874441 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.874443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.874749 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.874750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.874846 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.875242 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.875250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.877559 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
W20260619 20:50:31.878093 9579 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44327 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44327: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 20:50:31.880465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9449
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.882717 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.878401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:33665)
0619 20:50:31.878451 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.878462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.878504 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.878705 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.878706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.878811 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.878856 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.878857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.879418 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.879419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.879894 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.879895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.880182 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.880182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.880229 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.880645 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.880649 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.882635 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:50:31.882807 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.878549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33665 (local address 127.0.98.62:44327)
0619 20:50:31.878597 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.878599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.878606 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:50:31.878626 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.878627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.878635 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.878689 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.878915 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.879359 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.879954 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.880124 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.880169 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.882726 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33665: BlockingRecv error: recv got EOF from 127.0.98.3:33665 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:31 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:31.887538 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.883052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:33635)
0619 20:50:31.883086 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:31.883097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:31.883140 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:31.883245 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:31.883245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:31.883344 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:31.883367 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.883367 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.884110 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.884111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.884785 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:31.884787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.885251 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.885252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:31.885285 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.885657 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:31.885662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:31.887460 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 20:50:31.887604 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:31.883111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33635 (local address 127.0.98.62:44327)
0619 20:50:31.883155 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.883157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.883164 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:50:31.883183 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.883184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.883192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.883243 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.883437 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.884044 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.884868 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.885179 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.885247 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.887525 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33635: BlockingRecv error: recv got EOF from 127.0.98.3:33635 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 20:50:31.887763 9579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:50:31.888563 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:31.880888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.0.98.62:44327)
0619 20:50:31.880939 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:31.880942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:31.881099 (+ 157us) server_negotiation.cc:407] Connection header received
0619 20:50:31.881167 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:31.881168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:31.881181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:31.881250 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:31.882090 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.882807 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.883876 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:31.884060 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:31.884143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:31.885418 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:31.885425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:31.885425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:31.885438 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:31.887127 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:31.887478 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:31.887481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:31.887482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:31.887610 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:31.887876 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:31.887877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:31.887878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:31.888132 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:31.888247 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:31.888345 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:50:31.888427 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 20:50:32 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:32 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:32.646957 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:32.642411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:55241)
0619 20:50:32.642564 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:32.642576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:32.642642 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:32.642783 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:32.642784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:32.642934 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:32.642984 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:32.642985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.643604 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.643605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:32.644162 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:32.644163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.644517 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.644518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:32.644566 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:32.645132 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:32.645137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:32.646893 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
W20260619 20:50:32.647074 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:32.642474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55241 (local address 127.0.98.62:44327)
0619 20:50:32.642595 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:32.642597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:32.642626 (+ 29us) server_negotiation.cc:407] Connection header received
0619 20:50:32.642682 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:32.642683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:32.642691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:32.642760 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:32.643050 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.643535 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.644232 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.644420 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.644473 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:32.646976 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55241: BlockingRecv error: recv got EOF from 127.0.98.1:55241 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
Jun 19 20:50:32 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:32 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:32.769764 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:32.765024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:53333)
0619 20:50:32.765191 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:32.765211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:32.765296 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:32.765436 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:32.765437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:32.765620 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:32.765665 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:32.765667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.766331 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.766332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:32.766903 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:32.766904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.767319 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.767319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:32.767380 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:32.767861 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:32.767865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:32.769656 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
W20260619 20:50:32.769807 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:32.765125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53333 (local address 127.0.98.62:44327)
0619 20:50:32.765195 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:32.765197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:32.765281 (+ 84us) server_negotiation.cc:407] Connection header received
0619 20:50:32.765366 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:32.765367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:32.765380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:32.765431 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:32.765742 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.766228 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.767003 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.767222 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.767317 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:32.769742 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53333: BlockingRecv error: recv got EOF from 127.0.98.2:53333 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 20:50:32 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:32 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:32.893246 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:32.888214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:38475)
0619 20:50:32.888446 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:32.888470 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:32.888574 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:32.888754 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:32.888755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:32.888959 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:32.889028 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:32.889030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.889828 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.889829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:32.890385 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:32.890387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.890795 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.890796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:32.890854 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:32.891327 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:32.891331 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:32.893176 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
W20260619 20:50:32.893399 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:32.888333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38475 (local address 127.0.98.62:44327)
0619 20:50:32.888432 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:32.888435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:32.888551 (+ 116us) server_negotiation.cc:407] Connection header received
0619 20:50:32.888647 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:32.888648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:32.888664 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:32.888754 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:32.889122 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.889723 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.890500 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:32.890698 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:32.890794 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:32.893266 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38475: BlockingRecv error: recv got EOF from 127.0.98.3:38475 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 20:50:33 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:33 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:33.651872 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:33.647433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:34739)
0619 20:50:33.647555 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:33.647567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:33.647630 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:33.647784 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:33.647785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:33.647994 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:33.648044 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:33.648046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.648704 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.648704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:33.649223 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:33.649224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.649603 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.649604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:33.649643 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:33.650128 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:33.650132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:33.651816 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
W20260619 20:50:33.652046 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:33.647486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34739 (local address 127.0.98.62:44327)
0619 20:50:33.647618 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:33.647620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:33.647632 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:50:33.647684 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:33.647685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:33.647697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:33.647773 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:33.648117 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.648628 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.649324 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.649533 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.649596 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:33.651964 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34739: BlockingRecv error: recv got EOF from 127.0.98.1:34739 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
Jun 19 20:50:33 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:33 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:33.774834 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:33.770272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:37151)
0619 20:50:33.770385 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:33.770403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:33.770449 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:33.770583 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:33.770583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:33.770741 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:33.770789 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:33.770790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.771576 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.771577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:33.772225 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:33.772226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.772595 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.772595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:33.772633 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:33.773104 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:33.773108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:33.774778 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260619 20:50:33.774940 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:33.770335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37151 (local address 127.0.98.62:44327)
0619 20:50:33.770432 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:33.770434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:33.770445 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:50:33.770508 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:33.770508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:33.770518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:33.770579 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:33.770846 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.771480 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.772301 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.772502 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.772581 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:33.774865 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37151: BlockingRecv error: recv got EOF from 127.0.98.2:37151 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 20:50:33 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:33 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:33.898550 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:33.893734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:41879)
0619 20:50:33.893899 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:33.893912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:33.893984 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:33.894186 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:33.894187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:33.894374 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:33.894428 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:33.894430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.895067 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.895068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:33.895668 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:33.895670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.895985 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.895986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:33.896026 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:33.896581 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:33.896585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:33.898489 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
W20260619 20:50:33.898681 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:33.893872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41879 (local address 127.0.98.62:44327)
0619 20:50:33.893930 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:33.893932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:33.893975 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:50:33.894081 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:33.894082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:33.894097 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:33.894152 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:33.894494 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.894989 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.895755 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:33.895924 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:33.895970 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:33.898569 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41879: BlockingRecv error: recv got EOF from 127.0.98.3:41879 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 20:50:34 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:34 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:34.657114 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:34.652377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:44883)
0619 20:50:34.652519 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:34.652531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:34.652589 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:34.652726 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:34.652726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:34.652889 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:34.652940 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:34.652941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.653746 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.653748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:34.654288 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:34.654289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.654681 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.654682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:34.654767 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:34.655235 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:34.655239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:34.657014 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
W20260619 20:50:34.657188 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:34.652431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44883 (local address 127.0.98.62:44327)
0619 20:50:34.652566 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:34.652568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:34.652591 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:50:34.652646 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:34.652647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:34.652657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:34.652705 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:34.653002 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.653644 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.654375 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.654615 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.654667 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:34.657113 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44883: BlockingRecv error: recv got EOF from 127.0.98.1:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
Jun 19 20:50:34 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:34 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:34.779928 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:34.775299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:45861)
0619 20:50:34.775473 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:34.775486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:34.775570 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:34.775711 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:34.775711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:34.775884 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:34.775929 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:34.775931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.776581 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.776583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:34.777099 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:34.777101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.777568 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.777569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:34.777647 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:34.778044 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:34.778049 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:34.779828 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
W20260619 20:50:34.779960 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:34.775389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45861 (local address 127.0.98.62:44327)
0619 20:50:34.775477 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:34.775479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:34.775551 (+ 72us) server_negotiation.cc:407] Connection header received
0619 20:50:34.775642 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:34.775643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:34.775655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:34.775705 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:34.776014 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.776500 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.777193 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.777497 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.777570 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:34.779900 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45861: BlockingRecv error: recv got EOF from 127.0.98.2:45861 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 20:50:34 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:34 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:34.903978 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:34.899017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:50683)
0619 20:50:34.899182 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:34.899192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:34.899246 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:34.899367 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:34.899368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:34.899523 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:34.899585 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:34.899586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.900182 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.900183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:34.901205 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:34.901207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.901632 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.901633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:34.901689 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:34.902194 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:34.902199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:34.903878 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
W20260619 20:50:34.904016 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:34.899110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50683 (local address 127.0.98.62:44327)
0619 20:50:34.899174 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:34.899176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:34.899248 (+ 72us) server_negotiation.cc:407] Connection header received
0619 20:50:34.899298 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:34.899299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:34.899310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:34.899350 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:34.899648 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.900119 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.901350 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:34.901544 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:34.901630 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:34.903955 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50683: BlockingRecv error: recv got EOF from 127.0.98.3:50683 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 20:50:35 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:35 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:35.662197 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:35.657574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:44325)
0619 20:50:35.657744 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:35.657755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:35.657826 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:35.658033 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:35.658034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:35.658193 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:35.658245 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:35.658246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.658963 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.658965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:35.659499 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:35.659500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.659845 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.659846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:35.659884 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:35.660338 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:35.660343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:35.662139 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
W20260619 20:50:35.662357 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:35.657624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44325 (local address 127.0.98.62:44327)
0619 20:50:35.657765 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:35.657767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:35.657866 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:50:35.657892 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:35.657893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:35.657905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:35.657987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:35.658356 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.658882 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.659581 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.659785 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.659834 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:35.662264 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44325: BlockingRecv error: recv got EOF from 127.0.98.1:44325 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
Jun 19 20:50:35 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:35 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:35.788316 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:35.780463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:54321)
0619 20:50:35.780585 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:35.780597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:35.780661 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:35.780839 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:35.780840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:35.781013 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:35.781062 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:35.781064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.781778 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.781779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:35.782399 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:35.782401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.782768 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.782769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:35.782814 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:35.786407 (+ 3593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:35.786412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:35.788256 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
W20260619 20:50:35.788448 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:35.780540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54321 (local address 127.0.98.62:44327)
0619 20:50:35.780609 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:35.780611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:35.780646 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:50:35.780715 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:35.780716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:35.780726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:35.780785 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:35.781124 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.781708 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.782478 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.782703 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.782758 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:35.788367 (+ 5609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54321: BlockingRecv error: recv got EOF from 127.0.98.2:54321 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 20:50:35 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:35 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:35.909127 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:35.904507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:32983)
0619 20:50:35.904648 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:35.904659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:35.904746 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:35.904921 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:35.904922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:35.905133 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:35.905189 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:35.905190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.905848 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.905849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:35.906373 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:35.906374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.906699 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.906700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:35.906742 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:35.907247 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:35.907256 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:35.909046 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
W20260619 20:50:35.909264 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:35.904566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32983 (local address 127.0.98.62:44327)
0619 20:50:35.904634 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:35.904636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:35.904715 (+ 79us) server_negotiation.cc:407] Connection header received
0619 20:50:35.904823 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:35.904825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:35.904840 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:35.904897 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:35.905256 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.905779 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.906460 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:35.906635 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:35.906688 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:35.909151 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32983: BlockingRecv error: recv got EOF from 127.0.98.3:32983 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 20:50:36 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:36 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:36.667101 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:36.662708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:43205)
0619 20:50:36.662845 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:36.662856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:36.662914 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:36.663048 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:36.663049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:36.663203 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:36.663253 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:36.663254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.663924 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.663924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:36.664434 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:36.664435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.664770 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.664771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:36.664817 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:36.665267 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:36.665272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:36.667009 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260619 20:50:36.667150 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:36.662773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43205 (local address 127.0.98.62:44327)
0619 20:50:36.662908 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:36.662910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:36.662921 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:50:36.662957 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:36.662957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:36.662968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:36.663036 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:36.663335 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.663855 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.664499 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.664708 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.664760 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:36.667081 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43205: BlockingRecv error: recv got EOF from 127.0.98.1:43205 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
Jun 19 20:50:36 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:36 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:36.793344 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:36.788715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:51869)
0619 20:50:36.788844 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:36.788863 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:36.788939 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:36.789074 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:36.789075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:36.789253 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:36.789317 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:36.789320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.789968 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.789969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:36.790476 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:36.790477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.790896 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.790897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:36.790959 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:36.791439 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:36.791444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:36.793221 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":86,"threads_started":1}
W20260619 20:50:36.793390 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:36.788871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51869 (local address 127.0.98.62:44327)
0619 20:50:36.788938 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:36.788940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:36.788950 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:50:36.789001 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:36.789001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:36.789011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:36.789069 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:36.789400 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.789889 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.790557 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.790788 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.790896 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:36.793326 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51869: BlockingRecv error: recv got EOF from 127.0.98.2:51869 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 20:50:36 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:36 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:36.914005 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:36.909664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:57161)
0619 20:50:36.909849 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:36.909860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:36.909915 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:36.910046 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:36.910046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:36.910202 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:36.910252 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:36.910254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.910932 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.910932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:36.911444 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:36.911445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.911805 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.911806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:36.911842 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:36.912287 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:36.912291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:36.913941 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":116,"threads_started":1}
W20260619 20:50:36.914100 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:36.909738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57161 (local address 127.0.98.62:44327)
0619 20:50:36.909813 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:36.909815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:36.909918 (+ 103us) server_negotiation.cc:407] Connection header received
0619 20:50:36.909965 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:36.909966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:36.909979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:36.910031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:36.910323 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.910862 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.911545 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:36.911729 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:36.911792 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:36.914031 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57161: BlockingRecv error: recv got EOF from 127.0.98.3:57161 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 20:50:37 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:37 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:37.671934 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:37.667570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:46801)
0619 20:50:37.667726 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:37.667739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:37.667821 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:37.667985 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:37.667986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:37.668138 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:37.668183 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:37.668185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.668840 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.668841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:37.669436 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:37.669438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.669786 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.669786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:37.669822 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:37.670237 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:37.670241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:37.671870 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260619 20:50:37.672032 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:37.667590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46801 (local address 127.0.98.62:44327)
0619 20:50:37.667730 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:37.667732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:37.667789 (+ 57us) server_negotiation.cc:407] Connection header received
0619 20:50:37.667893 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:37.667894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:37.667905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:37.667971 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:37.668259 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.668779 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.669508 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.669725 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.669777 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:37.671961 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46801: BlockingRecv error: recv got EOF from 127.0.98.1:46801 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":53,"threads_started":1}
Jun 19 20:50:37 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:37 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:37.798512 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:37.793924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:44235)
0619 20:50:37.794074 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:37.794085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:37.794146 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:37.794272 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:37.794272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:37.794432 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:37.794479 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:37.794480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.795101 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.795102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:37.795683 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:37.795685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.795989 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.795990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:37.796028 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:37.796444 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:37.796448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:37.798445 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
W20260619 20:50:37.798588 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:37.794005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44235 (local address 127.0.98.62:44327)
0619 20:50:37.794079 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:37.794080 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:37.794133 (+ 53us) server_negotiation.cc:407] Connection header received
0619 20:50:37.794197 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:37.794197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:37.794208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:37.794264 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:37.794541 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.795011 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.795759 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.795930 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.795977 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:37.798532 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44235: BlockingRecv error: recv got EOF from 127.0.98.2:44235 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 20:50:37 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:37 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:37.919409 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:37.914578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:52617)
0619 20:50:37.914779 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:37.914791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:37.914879 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:37.915034 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:37.915035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:37.915225 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:37.915271 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:37.915272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.916039 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.916040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:37.916634 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:37.916636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.917021 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.917022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:37.917075 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:37.917613 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:37.917618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:37.919327 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":114,"threads_started":1}
W20260619 20:50:37.919528 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:37.914699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52617 (local address 127.0.98.62:44327)
0619 20:50:37.914766 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:37.914768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:37.914859 (+ 91us) server_negotiation.cc:407] Connection header received
0619 20:50:37.914942 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:37.914943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:37.914959 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:37.915028 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:37.915348 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.915953 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.916735 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:37.916936 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:37.917038 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:37.919421 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52617: BlockingRecv error: recv got EOF from 127.0.98.3:52617 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 20:50:38 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:38 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:38.677157 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:38.672416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:34735)
0619 20:50:38.672580 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:38.672593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:38.672658 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:38.672807 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:38.672808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:38.672960 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:38.673020 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:38.673022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.673720 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.673721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:38.674285 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:38.674286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.674794 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.674795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:38.674837 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:38.675348 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:38.675353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:38.677096 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
W20260619 20:50:38.677263 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:38.672471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34735 (local address 127.0.98.62:44327)
0619 20:50:38.672636 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:38.672638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:38.672646 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:38.672703 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:38.672703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:38.672712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:38.672768 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:38.673084 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.673656 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.674433 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.674727 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.674780 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:38.677189 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34735: BlockingRecv error: recv got EOF from 127.0.98.1:34735 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
Jun 19 20:50:38 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:38 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:38.804189 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:38.799174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:34367)
0619 20:50:38.799366 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:38.799383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:38.799459 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:38.799615 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:38.799616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:38.799809 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:38.799856 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:38.799858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.800528 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.800529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:38.801185 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:38.801187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.801588 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.801589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:38.801649 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:38.802130 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:38.802134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:38.804092 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
W20260619 20:50:38.804244 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:38.799255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34367 (local address 127.0.98.62:44327)
0619 20:50:38.799347 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:38.799349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:38.799440 (+ 91us) server_negotiation.cc:407] Connection header received
0619 20:50:38.799531 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:38.799532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:38.799545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:38.799619 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:38.799934 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.800448 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.801268 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.801504 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.801588 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:38.804179 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34367: BlockingRecv error: recv got EOF from 127.0.98.2:34367 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 20:50:38 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:38 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:38.924850 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:38.919916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:46533)
0619 20:50:38.920115 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:38.920127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:38.920220 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:38.920421 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:38.920422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:38.920645 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:38.920706 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:38.920714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.921592 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.921593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:38.922114 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:38.922116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.922458 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.922460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:38.922512 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:38.922989 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:38.922993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:38.924774 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
W20260619 20:50:38.924998 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:38.920032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46533 (local address 127.0.98.62:44327)
0619 20:50:38.920109 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:38.920111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:38.920202 (+ 91us) server_negotiation.cc:407] Connection header received
0619 20:50:38.920310 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:38.920312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:38.920325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:38.920385 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:38.920930 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.921524 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.922192 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:38.922371 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:38.922423 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:38.924889 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46533: BlockingRecv error: recv got EOF from 127.0.98.3:46533 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 20:50:39 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:39 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:39.682127 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:39.677656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:58467)
0619 20:50:39.677821 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:39.677834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:39.677895 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:39.678043 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:39.678044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:39.678197 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:39.678245 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:39.678246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.678881 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.678881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:39.679392 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:39.679393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.679711 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.679712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:39.679749 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:39.680160 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:39.680165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:39.682023 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260619 20:50:39.682219 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:39.677767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58467 (local address 127.0.98.62:44327)
0619 20:50:39.677906 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:39.677908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:39.677916 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:39.677936 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:39.677936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:39.677948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:39.678005 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:39.678307 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.678817 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.679457 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.679639 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.679700 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:39.682098 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58467: BlockingRecv error: recv got EOF from 127.0.98.1:58467 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
Jun 19 20:50:39 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:39 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:39.809602 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:39.804814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:36975)
0619 20:50:39.804944 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:39.804957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:39.805018 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:39.805170 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:39.805170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:39.805399 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:39.805462 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:39.805464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.806500 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.806501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:39.807001 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:39.807003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.807364 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.807365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:39.807400 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:39.807852 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:39.807856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:39.809546 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260619 20:50:39.809710 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:39.804871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36975 (local address 127.0.98.62:44327)
0619 20:50:39.804962 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:39.804963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:39.805022 (+ 59us) server_negotiation.cc:407] Connection header received
0619 20:50:39.805061 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:39.805061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:39.805092 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:39.805159 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:39.805566 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.806431 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.807098 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.807301 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.807354 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:39.809611 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36975: BlockingRecv error: recv got EOF from 127.0.98.2:36975 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 20:50:39 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:39 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:39.929921 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:39.925333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:40865)
0619 20:50:39.925491 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:39.925506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:39.925581 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:39.925733 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:39.925733 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:39.925916 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:39.925971 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:39.925973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.926717 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.926718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:39.927233 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:39.927234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.927614 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.927615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:39.927680 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:39.928117 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:39.928121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:39.929865 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
W20260619 20:50:39.930069 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:39.925389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40865 (local address 127.0.98.62:44327)
0619 20:50:39.925457 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:39.925460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:39.925562 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:50:39.925659 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:39.925660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:39.925676 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:39.925728 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:39.926055 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.926619 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.927316 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:39.927513 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:39.927605 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:39.929987 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40865: BlockingRecv error: recv got EOF from 127.0.98.3:40865 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 20:50:40 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:40 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:40.687358 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:40.682759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:53145)
0619 20:50:40.682897 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:40.682925 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:40.682989 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:40.683142 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:40.683143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:40.683338 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:40.683390 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:40.683392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.684063 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.684063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:40.684602 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:40.684604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.684969 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.684970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:40.685020 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:40.685580 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:40.685591 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:40.687300 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260619 20:50:40.687500 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:40.682821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53145 (local address 127.0.98.62:44327)
0619 20:50:40.682957 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:40.682959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:40.682970 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:50:40.683049 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:40.683050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:40.683060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:40.683122 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:40.683469 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.683995 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.684672 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.684860 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.684920 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:40.687410 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53145: BlockingRecv error: recv got EOF from 127.0.98.1:53145 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Jun 19 20:50:40 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:40 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:40.818471 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:40.810080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:47423)
0619 20:50:40.810261 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:40.810275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:40.810345 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:40.810492 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:40.810493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:40.810651 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:40.810698 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:40.810700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.811320 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.811321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:40.811844 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:40.811845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.812159 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.812160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:40.812198 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:40.816579 (+ 4381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:40.816584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:40.818413 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
W20260619 20:50:40.818590 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:40.810239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47423 (local address 127.0.98.62:44327)
0619 20:50:40.810307 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:40.810309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:40.810329 (+ 20us) server_negotiation.cc:407] Connection header received
0619 20:50:40.810409 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:40.810410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:40.810420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:40.810471 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:40.810764 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.811256 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.811915 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.812100 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.812149 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:40.818514 (+ 6365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47423: BlockingRecv error: recv got EOF from 127.0.98.2:47423 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 20:50:40 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:40 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:40.935657 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:40.930406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:46489)
0619 20:50:40.930610 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:40.930623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:40.930725 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:40.930976 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:40.930977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:40.931171 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:40.931221 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:40.931223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.931955 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.931956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:40.932608 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:40.932611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.933135 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.933136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:40.933181 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:40.933714 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:40.933719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:40.935598 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
W20260619 20:50:40.935783 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:40.930515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46489 (local address 127.0.98.62:44327)
0619 20:50:40.930609 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:40.930611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:40.930705 (+ 94us) server_negotiation.cc:407] Connection header received
0619 20:50:40.930858 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:40.930859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:40.930876 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:40.930951 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:40.931339 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.931860 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.932722 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:40.933051 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:40.933121 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:40.935684 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46489: BlockingRecv error: recv got EOF from 127.0.98.3:46489 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 20:50:41 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:41 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:41.692162 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:41.687875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:41659)
0619 20:50:41.688039 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:41.688051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:41.688111 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:41.688239 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:41.688239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:41.688427 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:41.688473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:41.688474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.689106 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.689107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:41.689650 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:41.689651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.689971 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.689972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:41.690008 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:41.690424 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:41.690428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:41.692106 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
W20260619 20:50:41.692273 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:41.687957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41659 (local address 127.0.98.62:44327)
0619 20:50:41.688093 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:41.688095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:41.688103 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:50:41.688163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:41.688164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:41.688173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:41.688220 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:41.688546 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.689044 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.689717 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.689910 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.689961 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:41.692193 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41659: BlockingRecv error: recv got EOF from 127.0.98.1:41659 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
Jun 19 20:50:41 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:41 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:41.823714 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:41.818962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:42535)
0619 20:50:41.819136 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:41.819150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:41.819228 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:41.819406 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:41.819407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:41.819618 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:41.819668 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:41.819670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.820384 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.820385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:41.820940 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:41.820942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.821445 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.821447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:41.821509 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:41.821946 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:41.821951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:41.823649 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
W20260619 20:50:41.823782 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:41.819056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42535 (local address 127.0.98.62:44327)
0619 20:50:41.819139 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:41.819141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:41.819258 (+ 117us) server_negotiation.cc:407] Connection header received
0619 20:50:41.819291 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:41.819292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:41.819306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:41.819387 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:41.819770 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.820289 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.821048 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.821343 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.821446 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:41.823737 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42535: BlockingRecv error: recv got EOF from 127.0.98.2:42535 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 20:50:41 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:41 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:41.940577 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:41.936130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:35117)
0619 20:50:41.936306 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:41.936317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:41.936389 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:41.936511 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:41.936512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:41.936664 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:41.936710 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:41.936712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.937415 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.937416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:41.937904 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:41.937905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.938212 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.938212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:41.938247 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:41.938731 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:41.938735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:41.940520 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
W20260619 20:50:41.940725 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:41.936271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35117 (local address 127.0.98.62:44327)
0619 20:50:41.936335 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:41.936336 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:41.936375 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:50:41.936440 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:41.936441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:41.936452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:41.936489 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:41.936799 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.937350 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.937977 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:41.938152 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:41.938203 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:41.940637 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35117: BlockingRecv error: recv got EOF from 127.0.98.3:35117 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 20:50:42 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:42 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:42.697783 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:42.692723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:51519)
0619 20:50:42.692898 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:42.692908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:42.693015 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:42.693189 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:42.693189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:42.693401 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:42.693450 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:42.693451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.694275 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.694275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:42.694823 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:42.694825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.695204 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.695205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:42.695241 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:42.695669 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:42.695673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:42.697727 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":117,"threads_started":1}
W20260619 20:50:42.697911 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:42.692811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51519 (local address 127.0.98.62:44327)
0619 20:50:42.692917 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:42.692919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:42.693015 (+ 96us) server_negotiation.cc:407] Connection header received
0619 20:50:42.693077 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:42.693078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:42.693096 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:42.693176 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:42.693517 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.694203 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.694914 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.695114 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.695197 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:42.697822 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51519: BlockingRecv error: recv got EOF from 127.0.98.1:51519 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
Jun 19 20:50:42 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:42 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:42.829103 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:42.824261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:52797)
0619 20:50:42.824403 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:42.824414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:42.824484 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:42.824634 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:42.824635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:42.824828 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:42.824873 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:42.824874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.825647 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.825648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:42.826408 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:42.826411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.826777 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.826778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:42.826833 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:42.827311 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:42.827316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:42.829009 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
W20260619 20:50:42.829151 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:42.824335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52797 (local address 127.0.98.62:44327)
0619 20:50:42.824384 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:42.824386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:42.824502 (+ 116us) server_negotiation.cc:407] Connection header received
0619 20:50:42.824536 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:42.824537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:42.824547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:42.824622 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:42.824956 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.825574 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.826472 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.826685 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.826769 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:42.829085 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52797: BlockingRecv error: recv got EOF from 127.0.98.2:52797 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 20:50:42 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:42 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:42.945662 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:42.941064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:58271)
0619 20:50:42.941240 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:42.941251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:42.941390 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:42.941532 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:42.941532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:42.941696 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:42.941746 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:42.941747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.942396 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.942397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:42.942917 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:42.942919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.943237 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.943238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:42.943278 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:42.943717 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:42.943722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:42.945563 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
W20260619 20:50:42.945726 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:42.941144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58271 (local address 127.0.98.62:44327)
0619 20:50:42.941217 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:42.941219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:42.941371 (+ 152us) server_negotiation.cc:407] Connection header received
0619 20:50:42.941443 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:42.941448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:42.941461 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:42.941510 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:42.941809 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.942307 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.942997 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:42.943173 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:42.943222 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:42.945656 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58271: BlockingRecv error: recv got EOF from 127.0.98.3:58271 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 20:50:43 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:43 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:43.703035 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:43.698322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:44625)
0619 20:50:43.698465 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:43.698481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:43.698555 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:43.698775 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:43.698776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:43.698985 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:43.699042 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:43.699046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.699742 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.699742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:43.700243 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:43.700244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.700602 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.700603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:43.700638 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:43.701130 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:43.701135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:43.702956 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260619 20:50:43.703195 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:43.698407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44625 (local address 127.0.98.62:44327)
0619 20:50:43.698618 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:43.698620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:43.698633 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:50:43.698664 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:43.698664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:43.698680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:43.698757 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:43.699134 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.699669 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.700334 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.700538 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.700593 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:43.703096 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44625: BlockingRecv error: recv got EOF from 127.0.98.1:44625 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":141,"threads_started":1}
Jun 19 20:50:43 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:43 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:43.833938 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:43.829571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:55193)
0619 20:50:43.829687 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:43.829699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:43.829757 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:43.829864 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:43.829865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:43.830021 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:43.830071 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:43.830072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.830693 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.830694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:43.831211 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:43.831213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.831521 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.831522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:43.831561 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:43.831971 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:43.831975 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:43.833868 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
W20260619 20:50:43.834005 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:43.829635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55193 (local address 127.0.98.62:44327)
0619 20:50:43.829703 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:43.829705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:43.829751 (+ 46us) server_negotiation.cc:407] Connection header received
0619 20:50:43.829800 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:43.829800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:43.829810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:43.829861 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:43.830128 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.830625 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.831281 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.831459 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.831514 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:43.833960 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55193: BlockingRecv error: recv got EOF from 127.0.98.2:55193 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 20:50:43 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:43 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:43.950956 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:43.946174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:60191)
0619 20:50:43.946397 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:43.946407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:43.946463 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:43.946579 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:43.946580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:43.946769 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:43.946821 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:43.946822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.947533 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.947534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:43.948047 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:43.948113 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.948464 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.948465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:43.948529 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:43.949020 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:43.949026 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:43.950844 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":136,"threads_started":1}
W20260619 20:50:43.950991 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:43.946298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60191 (local address 127.0.98.62:44327)
0619 20:50:43.946369 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:43.946371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:43.946463 (+ 92us) server_negotiation.cc:407] Connection header received
0619 20:50:43.946507 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:43.946508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:43.946519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:43.946567 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:43.946890 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.947445 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.948190 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:43.948382 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:43.948463 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:43.950924 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60191: BlockingRecv error: recv got EOF from 127.0.98.3:60191 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 20:50:44 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:44 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:44.708339 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:44.703661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:45169)
0619 20:50:44.703881 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:44.703898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:44.703982 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:44.704157 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:44.704158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:44.704391 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:44.704440 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:44.704442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.705082 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.705084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:44.705754 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:44.705756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.706096 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.706097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:44.706136 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:44.706578 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:44.706583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:44.708283 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":124,"threads_started":1}
W20260619 20:50:44.708480 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:44.703727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45169 (local address 127.0.98.62:44327)
0619 20:50:44.703904 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:44.703906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:44.703957 (+ 51us) server_negotiation.cc:407] Connection header received
0619 20:50:44.704052 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:44.704052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:44.704065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:44.704130 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:44.704513 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.705001 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.705835 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.706033 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.706084 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:44.708393 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45169: BlockingRecv error: recv got EOF from 127.0.98.1:45169 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
Jun 19 20:50:44 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:44 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:44.838842 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:44.834467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:60963)
0619 20:50:44.834645 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:44.834689 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:44.834741 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:44.834890 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:44.834891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:44.835072 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:44.835122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:44.835123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.835759 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.835760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:44.836298 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:44.836300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.836605 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.836606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:44.836646 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:44.837059 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:44.837063 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:44.838768 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
W20260619 20:50:44.838907 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:44.834688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60963 (local address 127.0.98.62:44327)
0619 20:50:44.834768 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:44.834770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:44.834777 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:50:44.834801 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:44.834802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:44.834816 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:44.834867 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:44.835192 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.835673 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.836380 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.836548 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.836602 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:44.838862 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60963: BlockingRecv error: recv got EOF from 127.0.98.2:60963 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 20:50:44 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:44 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:44.956303 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:44.951484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:41249)
0619 20:50:44.951642 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:44.951654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:44.951719 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:44.951847 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:44.951848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:44.952023 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:44.952072 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:44.952073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.952868 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.952869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:44.953549 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:44.953551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.953991 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.953992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:44.954033 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:44.954446 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:44.954451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:44.956247 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
W20260619 20:50:44.956470 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:44.951547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41249 (local address 127.0.98.62:44327)
0619 20:50:44.951642 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:44.951644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:44.951702 (+ 58us) server_negotiation.cc:407] Connection header received
0619 20:50:44.951773 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:44.951774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:44.951786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:44.951826 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:44.952131 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.952758 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.953667 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:44.953887 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:44.953956 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:44.956355 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41249: BlockingRecv error: recv got EOF from 127.0.98.3:41249 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 20:50:45 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:45 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:45.713150 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:45.708785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:58959)
0619 20:50:45.708931 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:45.708943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:45.708996 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:45.709155 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:45.709155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:45.709347 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:45.709395 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:45.709397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.710013 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.710022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:45.710513 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:45.710515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.710847 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.710848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:45.710885 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:45.711365 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:45.711376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:45.713093 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260619 20:50:45.713266 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:45.708877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58959 (local address 127.0.98.62:44327)
0619 20:50:45.709030 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:45.709031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:45.709041 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:50:45.709062 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:45.709063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:45.709074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:45.709135 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:45.709461 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.709948 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.710580 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.710760 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.710838 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:45.713176 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58959: BlockingRecv error: recv got EOF from 127.0.98.1:58959 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
Jun 19 20:50:45 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:45 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:45.847870 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:45.839379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:45971)
0619 20:50:45.839539 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:45.839550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:45.839612 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:45.839747 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:45.839748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:45.839905 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:45.839952 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:45.839954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.840744 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.840745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:45.841263 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:45.841265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.841603 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.841604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:45.841646 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:45.845929 (+ 4283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:45.845934 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:45.847813 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
W20260619 20:50:45.848001 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:45.839481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45971 (local address 127.0.98.62:44327)
0619 20:50:45.839554 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:45.839556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:45.839597 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:50:45.839665 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:45.839666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:45.839676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:45.839717 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:45.840127 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.840658 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.841353 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.841533 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.841581 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:45.847924 (+ 6343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45971: BlockingRecv error: recv got EOF from 127.0.98.2:45971 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 20:50:45 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:45 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:45.961593 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:45.956804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.3:54617)
0619 20:50:45.957005 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:45.957018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:45.957118 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:45.957270 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:45.957270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:45.957483 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:45.957531 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:45.957534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.958240 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.958242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:45.958781 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:45.958782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.959175 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.959176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:45.959252 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:45.959711 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:45.959715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:45.961525 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
W20260619 20:50:45.961688 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:45.956908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54617 (local address 127.0.98.62:44327)
0619 20:50:45.956999 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:45.957002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:45.957099 (+ 97us) server_negotiation.cc:407] Connection header received
0619 20:50:45.957176 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:45.957177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:45.957192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:45.957263 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:45.957620 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.958145 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.958897 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:45.959071 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:45.959157 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:45.961621 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54617: BlockingRecv error: recv got EOF from 127.0.98.3:54617 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 20:50:46 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:46 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:46.718537 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:46.713727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.1:60251)
0619 20:50:46.713872 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:46.713883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:46.713927 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:46.714118 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:46.714119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:46.714381 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:46.714456 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:46.714458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.715283 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.715284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:46.715806 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:46.715807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.716185 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.716186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:46.716224 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:46.716675 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:46.716679 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:46.718471 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260619 20:50:46.718664 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:46.713854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60251 (local address 127.0.98.62:44327)
0619 20:50:46.713983 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:46.713985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:46.713992 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:50:46.714012 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:46.714013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:46.714022 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:46.714110 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:46.714544 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.715185 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.715912 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.716115 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.716193 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:46.718560 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60251: BlockingRecv error: recv got EOF from 127.0.98.1:60251 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
Jun 19 20:50:46 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:50:46 dist-test-slave-0smx krb5kdc[8580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:50:46.853032 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:46.848393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44327 (local address 127.0.98.2:41433)
0619 20:50:46.848550 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:46.848562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:46.848628 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:46.848771 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:46.848772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:46.848953 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:46.848997 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:46.848999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.849701 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.849702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:46.850243 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:46.850245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.850631 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.850632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:46.850690 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:46.851148 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:46.851153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:46.852934 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
W20260619 20:50:46.853061 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:46.848468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41433 (local address 127.0.98.62:44327)
0619 20:50:46.848539 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:46.848541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:46.848609 (+ 68us) server_negotiation.cc:407] Connection header received
0619 20:50:46.848683 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:46.848683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:46.848694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:46.848764 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:46.849076 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.849618 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.850338 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:46.850529 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:46.850633 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:46.853001 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41433: BlockingRecv error: recv got EOF from 127.0.98.2:41433 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:50:46.886395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9164
I20260619 20:50:46.890861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9315
I20260619 20:50:46.895059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9449
I20260619 20:50:46.899006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9093
2026-06-19T20:50:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17412 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T20:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:46Z Disabled control of system clock
I20260619 20:50:46.925621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39367
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 20:50:47.004815 9645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.005007 9645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.005025 9645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.006198 9645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:47.006239 9645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.006251 9645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.006263 9645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:47.006274 9645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:47.007476 9645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.007670 9645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.007890 9645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.012984 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.013060 9645 server_base.cc:1034] running on GCE node
W20260619 20:50:47.012992 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.012996 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.013686 9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.013993 9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.015157 9645 hybrid_clock.cc:648] HybridClock initialized: now 1781902247015146 us; error 31 us; skew 500 ppm
I20260619 20:50:47.020318 9645 webserver.cc:460] Webserver started at http://127.0.98.62:33029/ using document root <none> and password file <none>
I20260619 20:50:47.020567 9645 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.020638 9645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.020794 9645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.021981 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "4b42fe56590d4615937460124fb09723"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "55be14e1d58df5d3d59334b281425b78"
server_key_iv: "b225f36c22bac69d1c71f9bcbb863de3"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.022353 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b42fe56590d4615937460124fb09723"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "55be14e1d58df5d3d59334b281425b78"
server_key_iv: "b225f36c22bac69d1c71f9bcbb863de3"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.023670 9645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:47.024401 9659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.024567 9645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:47.024667 9645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "4b42fe56590d4615937460124fb09723"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "55be14e1d58df5d3d59334b281425b78"
server_key_iv: "b225f36c22bac69d1c71f9bcbb863de3"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.024745 9645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.039091 9645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.039438 9645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.039587 9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.043464 9645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39367
I20260619 20:50:47.043497 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39367 every 8 connection(s)
I20260619 20:50:47.043820 9645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:50:47.044473 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:47.047021 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723: Bootstrap starting.
I20260619 20:50:47.047587 9712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:47.047865 9712 log.cc:824] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:47.048645 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723: No bootstrap required, opened a new log
I20260619 20:50:47.049556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9645
I20260619 20:50:47.049659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:50:47.049839 392 external_mini_cluster.cc:1428] Setting key 7f943ecbffa7dff9ffb91e98ab687152
I20260619 20:50:47.050498 9712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b42fe56590d4615937460124fb09723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260619 20:50:47.050675 9712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:47.050709 9712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b42fe56590d4615937460124fb09723, State: Initialized, Role: FOLLOWER
I20260619 20:50:47.050838 9712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b42fe56590d4615937460124fb09723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260619 20:50:47.050915 9712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:50:47.050949 9712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:50:47.051000 9712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:47.051625 9712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b42fe56590d4615937460124fb09723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260619 20:50:47.051707 9712 leader_election.cc:302] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b42fe56590d4615937460124fb09723; no voters:
I20260619 20:50:47.051893 9712 leader_election.cc:288] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:50:47.051931 9717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:47.052047 9717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [term 1 LEADER]: Becoming Leader. State: Replica: 4b42fe56590d4615937460124fb09723, State: Running, Role: LEADER
I20260619 20:50:47.052155 9712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:50:47.052181 9717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b42fe56590d4615937460124fb09723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260619 20:50:47.052572 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b42fe56590d4615937460124fb09723. Latest consensus state: current_term: 1 leader_uuid: "4b42fe56590d4615937460124fb09723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b42fe56590d4615937460124fb09723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } } }
I20260619 20:50:47.052578 9719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b42fe56590d4615937460124fb09723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b42fe56590d4615937460124fb09723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } } }
I20260619 20:50:47.052783 9719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:47.052970 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:47.053351 9726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:50:47.053882 9726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:50:47.055250 9726 catalog_manager.cc:1347] Generated new cluster ID: 85c28c300a4f4737ba0217737cdb69c6
I20260619 20:50:47.055297 9726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:50:47.056286 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.050406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.0.98.62:39367)
0619 20:50:47.050598 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.050607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.050643 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:50:47.050767 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.050774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.050849 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.051012 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.051379 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.052914 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.054043 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.054661 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.054776 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.055631 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.055634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.055635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.055827 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.055865 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.055877 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.055949 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.056009 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.056081 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260619 20:50:47.064874 9726 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:50:47.065558 9726 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:50:47.074496 9726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b42fe56590d4615937460124fb09723: Generated new TSK 0
I20260619 20:50:47.074661 9726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:50:47.076731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 20:50:47.172430 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.172621 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.172641 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.173791 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.173830 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.173854 9736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:50:47.175202 9736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.175432 9736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.175740 9736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.181468 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.181511 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.181517 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.181716 9736 server_base.cc:1034] running on GCE node
I20260619 20:50:47.181912 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.182147 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.183302 9736 hybrid_clock.cc:648] HybridClock initialized: now 1781902247183294 us; error 35 us; skew 500 ppm
I20260619 20:50:47.184454 9736 webserver.cc:460] Webserver started at http://127.0.98.1:41991/ using document root <none> and password file <none>
I20260619 20:50:47.184675 9736 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.184741 9736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.184881 9736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.185972 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f4d3df383e844bea243371d294a3478"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "bde5e142b56ef7bbc61e1969f3b20eb0"
server_key_iv: "9e9cb0a70efcd10c88fc9a7a58e842b6"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.186326 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f4d3df383e844bea243371d294a3478"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "bde5e142b56ef7bbc61e1969f3b20eb0"
server_key_iv: "9e9cb0a70efcd10c88fc9a7a58e842b6"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.187611 9736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:47.188383 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.188541 9736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:47.188630 9736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "6f4d3df383e844bea243371d294a3478"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "bde5e142b56ef7bbc61e1969f3b20eb0"
server_key_iv: "9e9cb0a70efcd10c88fc9a7a58e842b6"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.188704 9736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.199880 9736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.200170 9736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.200332 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.200533 9736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:47.200855 9736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:47.200922 9736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.200961 9736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:47.200987 9736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.206619 9736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34865
I20260619 20:50:47.206692 9863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34865 every 8 connection(s)
I20260619 20:50:47.207001 9736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:50:47.211716 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.207667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32871 (local address 127.0.98.62:39367)
0619 20:50:47.207741 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.207744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.207963 (+ 219us) server_negotiation.cc:407] Connection header received
0619 20:50:47.208382 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.208383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.208393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.208443 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.209007 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.209561 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.210500 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.210693 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.210774 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.211479 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.211481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.211481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.211485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.211498 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.211503 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.211548 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.211618 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.211644 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:50:47.211850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9736
I20260619 20:50:47.211941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:50:47.211966 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.207638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.1:32871)
0619 20:50:47.207842 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:47.207872 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:47.208307 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:47.208491 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:47.208494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:47.208501 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:47.208932 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:47.208936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.209633 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.209633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:47.210398 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:47.210400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.210779 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.210780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:47.210882 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.211354 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:50:47.211354 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:47.211379 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:50:47.211402 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:47.211559 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:47.211562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:47.211564 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:47.211613 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:50:47.211716 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":121,"threads_started":1}
I20260619 20:50:47.212136 392 external_mini_cluster.cc:1428] Setting key 97cfcb689f44dd91ec343343d998249a
I20260619 20:50:47.212457 9864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39367
I20260619 20:50:47.212561 9864 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:47.212738 9864 heartbeater.cc:511] Master 127.0.98.62:39367 requested a full tablet report, sending...
I20260619 20:50:47.213094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 20:50:47.213253 9676 ts_manager.cc:194] Registered new tserver with Master: 6f4d3df383e844bea243371d294a3478 (127.0.98.1:34865)
I20260619 20:50:47.213846 9676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:32871
W20260619 20:50:47.292232 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.292462 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.292506 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.293705 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.293780 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.293823 9867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:50:47.295125 9867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.295384 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.295676 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.300834 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.300860 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.300906 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.300938 9867 server_base.cc:1034] running on GCE node
I20260619 20:50:47.301167 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.301455 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.302641 9867 hybrid_clock.cc:648] HybridClock initialized: now 1781902247302620 us; error 37 us; skew 500 ppm
I20260619 20:50:47.304133 9867 webserver.cc:460] Webserver started at http://127.0.98.2:45705/ using document root <none> and password file <none>
I20260619 20:50:47.304427 9867 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.304486 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.304673 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.305954 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "92cc5344b66949a2b461e527a4315fee"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "e92f0744883fe9d15ad3f665011f3509"
server_key_iv: "fccce8b9da5bc939400998b2dc922257"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.306363 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "92cc5344b66949a2b461e527a4315fee"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "e92f0744883fe9d15ad3f665011f3509"
server_key_iv: "fccce8b9da5bc939400998b2dc922257"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.307843 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:47.308537 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.308725 9867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:47.308826 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "92cc5344b66949a2b461e527a4315fee"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "e92f0744883fe9d15ad3f665011f3509"
server_key_iv: "fccce8b9da5bc939400998b2dc922257"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.308902 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.355331 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.355659 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.355819 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.356051 9867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:47.356411 9867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:47.356446 9867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.356468 9867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:47.356518 9867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.362141 9867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37549
I20260619 20:50:47.362192 9994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37549 every 8 connection(s)
I20260619 20:50:47.362510 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:50:47.367445 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.363243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44433 (local address 127.0.98.62:39367)
0619 20:50:47.363326 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.363328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.363548 (+ 220us) server_negotiation.cc:407] Connection header received
0619 20:50:47.364029 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.364030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.364043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.364101 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.364794 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.365283 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.366246 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.366423 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.366475 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.367221 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.367222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.367222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.367226 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.367242 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.367246 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.367288 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.367367 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.367388 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:50:47.367652 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.363215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.2:44433)
0619 20:50:47.363429 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:47.363462 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:47.363940 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:47.364174 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:47.364178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:47.364185 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:47.364690 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:47.364694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.365402 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.365403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:47.366169 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:47.366171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.366501 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.366502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:47.366608 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.367093 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:50:47.367094 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:47.367121 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:50:47.367141 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:47.367299 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:47.367302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:47.367304 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:47.367355 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:50:47.367439 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260619 20:50:47.368093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9867
I20260619 20:50:47.368183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:50:47.368201 9995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39367
I20260619 20:50:47.368291 9995 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:47.368443 392 external_mini_cluster.cc:1428] Setting key c3052d6ea215c3fb70f9dc4f2b351f23
I20260619 20:50:47.368490 9995 heartbeater.cc:511] Master 127.0.98.62:39367 requested a full tablet report, sending...
I20260619 20:50:47.369019 9675 ts_manager.cc:194] Registered new tserver with Master: 92cc5344b66949a2b461e527a4315fee (127.0.98.2:37549)
I20260619 20:50:47.369467 9675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44433
I20260619 20:50:47.369480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 20:50:47.448169 9998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.448401 9998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.448436 9998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.449606 9998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.449663 9998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.449705 9998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:50:47.450984 9998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.451220 9998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.451494 9998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.456562 10004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.456534 10006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.456579 10003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.456730 9998 server_base.cc:1034] running on GCE node
I20260619 20:50:47.456946 9998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.457218 9998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.458379 9998 hybrid_clock.cc:648] HybridClock initialized: now 1781902247458359 us; error 45 us; skew 500 ppm
I20260619 20:50:47.459556 9998 webserver.cc:460] Webserver started at http://127.0.98.3:43931/ using document root <none> and password file <none>
I20260619 20:50:47.459792 9998 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.459859 9998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.460003 9998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.461017 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "b8129d2f2f1449d48ec9d976e1eebe5d"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "201fcabfab92ef067b2f7b77b74c0981"
server_key_iv: "fca52e23de6434dcd513d9c9845d4ec9"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.461416 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b8129d2f2f1449d48ec9d976e1eebe5d"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "201fcabfab92ef067b2f7b77b74c0981"
server_key_iv: "fca52e23de6434dcd513d9c9845d4ec9"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.462679 9998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:47.463406 10012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.463557 9998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:47.463634 9998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "b8129d2f2f1449d48ec9d976e1eebe5d"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "201fcabfab92ef067b2f7b77b74c0981"
server_key_iv: "fca52e23de6434dcd513d9c9845d4ec9"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.463690 9998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.485966 9998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.486199 9998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.486294 9998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.486452 9998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:47.486773 9998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:47.486805 9998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.486860 9998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:47.486893 9998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.492693 9998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42337
I20260619 20:50:47.492740 10125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42337 every 8 connection(s)
I20260619 20:50:47.493108 9998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:50:47.493759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 9998
I20260619 20:50:47.493852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:50:47.494082 392 external_mini_cluster.cc:1428] Setting key 0a35e09581b8c52c5105515d9d6623ab
I20260619 20:50:47.498018 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.493829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39089 (local address 127.0.98.62:39367)
0619 20:50:47.493886 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.493888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.494129 (+ 241us) server_negotiation.cc:407] Connection header received
0619 20:50:47.494631 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.494632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.494651 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.494696 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.495352 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.495862 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.496767 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.496944 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.496994 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.497792 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.497793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.497794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.497797 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.497810 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.497814 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.497862 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.497939 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.497963 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:50:47.498162 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.493838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.3:39089)
0619 20:50:47.494013 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:47.494055 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:47.494556 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:47.494761 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:47.494765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:47.494775 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:47.495271 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:47.495276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.495937 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.495938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:47.496681 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:47.496683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:47.497022 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:47.497023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:47.497174 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:47.497657 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:50:47.497657 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:47.497683 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:50:47.497702 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:47.497873 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:47.497875 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:47.497877 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:50:47.497921 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:50:47.497988 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260619 20:50:47.498533 10126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39367
I20260619 20:50:47.498636 10126 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:47.498805 10126 heartbeater.cc:511] Master 127.0.98.62:39367 requested a full tablet report, sending...
I20260619 20:50:47.499102 9675 ts_manager.cc:194] Registered new tserver with Master: b8129d2f2f1449d48ec9d976e1eebe5d (127.0.98.3:42337)
I20260619 20:50:47.499418 9675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39089
I20260619 20:50:47.505728 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:50:47.507517 9716 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:47.507073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.98.62:39367)
0619 20:50:47.507126 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.507127 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.507177 (+ 50us) server_negotiation.cc:407] Connection header received
0619 20:50:47.507256 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.507257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.507266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.507327 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.507466 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47306: BlockingRecv error: recv got EOF from 127.0.0.1:47306 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:50:47.507604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9736
I20260619 20:50:47.512439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9867
I20260619 20:50:47.516320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9998
I20260619 20:50:47.520406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 9645
2026-06-19T20:50:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (620 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T20:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:47Z Disabled control of system clock
I20260619 20:50:47.546169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44413
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:50:47.624394 10137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.624580 10137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.624599 10137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.625653 10137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:47.625690 10137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.625703 10137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.625715 10137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:47.625733 10137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:47.627024 10137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.627200 10137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.627408 10137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.632153 10143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.632176 10145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.632194 10137 server_base.cc:1034] running on GCE node
W20260619 20:50:47.632165 10142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.632570 10137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.632865 10137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.634037 10137 hybrid_clock.cc:648] HybridClock initialized: now 1781902247634028 us; error 57 us; skew 500 ppm
I20260619 20:50:47.635223 10137 webserver.cc:460] Webserver started at http://127.0.98.62:34653/ using document root <none> and password file <none>
I20260619 20:50:47.635466 10137 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.635538 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.635679 10137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.636703 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "6402b6bc8c7f4919a5eb8483179302e9"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "806c1ea39f29465486350d70dcfea705"
server_key_iv: "2172eeaae9a4adba20774856c7f27501"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.637073 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "6402b6bc8c7f4919a5eb8483179302e9"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "806c1ea39f29465486350d70dcfea705"
server_key_iv: "2172eeaae9a4adba20774856c7f27501"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.638449 10137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:50:47.639245 10151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.639420 10137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:47.639518 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "6402b6bc8c7f4919a5eb8483179302e9"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "806c1ea39f29465486350d70dcfea705"
server_key_iv: "2172eeaae9a4adba20774856c7f27501"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.639611 10137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.642894 10137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.643159 10137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.643290 10137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.647014 10137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44413
I20260619 20:50:47.647068 10203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44413 every 8 connection(s)
I20260619 20:50:47.647359 10137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:50:47.647989 10204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:47.650035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10137
I20260619 20:50:47.650128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:50:47.650339 392 external_mini_cluster.cc:1428] Setting key aa463489b5036c7eac1f275af6d48d2f
I20260619 20:50:47.651050 10204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9: Bootstrap starting.
I20260619 20:50:47.651607 10204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:47.651958 10204 log.cc:824] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:47.652843 10204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9: No bootstrap required, opened a new log
I20260619 20:50:47.653012 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.650846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.0.98.62:44413)
0619 20:50:47.651054 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.651061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.651084 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:50:47.651513 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.651520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.651557 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.651730 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.652515 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.652519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.652519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.652524 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.652573 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.652586 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.652685 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.652721 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.652777 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260619 20:50:47.654433 10204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6402b6bc8c7f4919a5eb8483179302e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44413 } }
I20260619 20:50:47.654559 10204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:47.654582 10204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6402b6bc8c7f4919a5eb8483179302e9, State: Initialized, Role: FOLLOWER
I20260619 20:50:47.654659 10204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6402b6bc8c7f4919a5eb8483179302e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44413 } }
I20260619 20:50:47.655015 10204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:50:47.655063 10204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:50:47.655110 10204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:47.655606 10204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6402b6bc8c7f4919a5eb8483179302e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44413 } }
I20260619 20:50:47.655733 10204 leader_election.cc:302] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6402b6bc8c7f4919a5eb8483179302e9; no voters:
I20260619 20:50:47.655900 10204 leader_election.cc:288] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:50:47.655952 10209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:47.656138 10209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6402b6bc8c7f4919a5eb8483179302e9, State: Running, Role: LEADER
I20260619 20:50:47.656225 10204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:50:47.656231 10209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6402b6bc8c7f4919a5eb8483179302e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44413 } }
I20260619 20:50:47.656625 10211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6402b6bc8c7f4919a5eb8483179302e9. Latest consensus state: current_term: 1 leader_uuid: "6402b6bc8c7f4919a5eb8483179302e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6402b6bc8c7f4919a5eb8483179302e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44413 } } }
I20260619 20:50:47.656714 10211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:47.657016 10210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6402b6bc8c7f4919a5eb8483179302e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6402b6bc8c7f4919a5eb8483179302e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44413 } } }
I20260619 20:50:47.657145 10210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9 [sys.catalog]: This master's current role is: LEADER
W20260619 20:50:47.658177 10225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:50:47.658275 10225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:50:47.658365 10217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:50:47.658553 10217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:50:47.660257 10217 catalog_manager.cc:1347] Generated new cluster ID: a5ac0fc3900b491289beccc629b14681
I20260619 20:50:47.660313 10217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:50:47.667495 10217 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:50:47.667641 10217 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:50:47.673779 10217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6402b6bc8c7f4919a5eb8483179302e9: Generated new TSK 0
I20260619 20:50:47.673928 10217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:50:47.676373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44413
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:50:47.766183 10228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.766388 10228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.766410 10228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.767563 10228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.767602 10228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.767628 10228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:50:47.768949 10228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.769183 10228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.769543 10228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.774569 10236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.774590 10233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.774590 10234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.774889 10228 server_base.cc:1034] running on GCE node
I20260619 20:50:47.775086 10228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.775331 10228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.776491 10228 hybrid_clock.cc:648] HybridClock initialized: now 1781902247776476 us; error 31 us; skew 500 ppm
I20260619 20:50:47.777726 10228 webserver.cc:460] Webserver started at http://127.0.98.1:39197/ using document root <none> and password file <none>
I20260619 20:50:47.777972 10228 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.778044 10228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.778177 10228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.779207 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce7df59e92d94f8a8cc12c7bbe5b073f"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "c79923188bc5ac4657e1a3ef64890077"
server_key_iv: "536543f11fb65fc9f14c72bd21093073"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.779572 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce7df59e92d94f8a8cc12c7bbe5b073f"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "c79923188bc5ac4657e1a3ef64890077"
server_key_iv: "536543f11fb65fc9f14c72bd21093073"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.780922 10228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:50:47.781826 10242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.781997 10228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:50:47.782094 10228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "ce7df59e92d94f8a8cc12c7bbe5b073f"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "c79923188bc5ac4657e1a3ef64890077"
server_key_iv: "536543f11fb65fc9f14c72bd21093073"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.782171 10228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.785386 10228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.785645 10228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.785786 10228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.785996 10228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:47.786285 10228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:47.786352 10228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.786391 10228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:47.786415 10228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.792469 10228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44599
I20260619 20:50:47.792510 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44599 every 8 connection(s)
I20260619 20:50:47.792865 10228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:50:47.796002 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.793891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58867 (local address 127.0.98.62:44413)
0619 20:50:47.793931 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.793933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.793943 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:50:47.794517 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.794517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.794521 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.794576 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.795776 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.795777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.795777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.795781 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.795794 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.795798 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.795850 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.795939 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.795951 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:50:47.796284 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.793649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44413 (local address 127.0.98.1:58867)
0619 20:50:47.793824 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:47.793866 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:47.794423 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:47.794650 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:47.794654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:47.794663 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:47.795624 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:50:47.795625 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:47.795659 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:50:47.795693 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:47.795869 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:47.795872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:47.795875 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:47.795943 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:50:47.796006 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260619 20:50:47.796693 10356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44413
I20260619 20:50:47.796794 10356 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:47.796927 10356 heartbeater.cc:511] Master 127.0.98.62:44413 requested a full tablet report, sending...
I20260619 20:50:47.797443 10168 ts_manager.cc:194] Registered new tserver with Master: ce7df59e92d94f8a8cc12c7bbe5b073f (127.0.98.1:44599)
I20260619 20:50:47.801499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10228
I20260619 20:50:47.801566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:50:47.801779 392 external_mini_cluster.cc:1428] Setting key edb30932a1ef866c7dcb89c54ea32a5d
I20260619 20:50:47.802635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44413
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:50:47.882288 10359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.882479 10359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.882500 10359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.883658 10359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.883699 10359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.883724 10359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:50:47.885099 10359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.885360 10359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.885684 10359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:47.890506 10367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:47.890514 10365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.890534 10359 server_base.cc:1034] running on GCE node
W20260619 20:50:47.890514 10364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:47.890929 10359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:47.891162 10359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:47.892331 10359 hybrid_clock.cc:648] HybridClock initialized: now 1781902247892318 us; error 26 us; skew 500 ppm
I20260619 20:50:47.893572 10359 webserver.cc:460] Webserver started at http://127.0.98.2:38515/ using document root <none> and password file <none>
I20260619 20:50:47.893860 10359 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:47.893931 10359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:47.894057 10359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:47.895030 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3c349083f17468c9788484479597696"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "18890e2d9c013693f897b11b9a6ed87f"
server_key_iv: "308bc8b469dae042dbf2ddf14a8b0967"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.895306 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3c349083f17468c9788484479597696"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "18890e2d9c013693f897b11b9a6ed87f"
server_key_iv: "308bc8b469dae042dbf2ddf14a8b0967"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.896510 10359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:50:47.897195 10373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.897365 10359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:47.897460 10359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "e3c349083f17468c9788484479597696"
format_stamp: "Formatted at 2026-06-19 20:50:47 on dist-test-slave-0smx"
server_key: "18890e2d9c013693f897b11b9a6ed87f"
server_key_iv: "308bc8b469dae042dbf2ddf14a8b0967"
server_key_version: "encryptionkey@0"
I20260619 20:50:47.897548 10359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:47.900655 10359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:47.900913 10359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:47.901046 10359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:47.901275 10359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:47.901655 10359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:47.901714 10359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.901754 10359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:47.901782 10359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:47.907456 10359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32775
I20260619 20:50:47.907487 10486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32775 every 8 connection(s)
I20260619 20:50:47.907773 10359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:50:47.910606 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.908438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53331 (local address 127.0.98.62:44413)
0619 20:50:47.908479 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:47.908481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:47.908704 (+ 223us) server_negotiation.cc:407] Connection header received
0619 20:50:47.909194 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:47.909196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:47.909202 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:47.909264 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:47.910348 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:47.910349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:47.910350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:47.910354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:47.910368 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:47.910372 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:47.910447 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:47.910522 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 20:50:47.910536 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:50:47.910784 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:47.908424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44413 (local address 127.0.98.2:53331)
0619 20:50:47.908599 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:47.908627 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:47.909092 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:47.909349 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:47.909352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:47.909361 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:47.910195 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:50:47.910195 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:47.910220 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:50:47.910245 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:47.910444 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:47.910449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:47.910452 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:47.910531 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:50:47.910588 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260619 20:50:47.911249 10487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44413
I20260619 20:50:47.911319 10487 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:47.911464 10487 heartbeater.cc:511] Master 127.0.98.62:44413 requested a full tablet report, sending...
I20260619 20:50:47.911806 10168 ts_manager.cc:194] Registered new tserver with Master: e3c349083f17468c9788484479597696 (127.0.98.2:32775)
I20260619 20:50:47.917004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10359
I20260619 20:50:47.917078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:50:47.917251 392 external_mini_cluster.cc:1428] Setting key 32a32407b62b1cb9d2bd9b31b044f255
I20260619 20:50:47.918145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44413
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:50:47.995738 10490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:47.995966 10490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:47.995999 10490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:47.997087 10490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:47.997151 10490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:47.997192 10490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:50:47.998487 10490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:47.998736 10490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:47.999014 10490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:48.003996 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:48.004069 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:48.004132 10490 server_base.cc:1034] running on GCE node
W20260619 20:50:48.004069 10498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:48.004442 10490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:48.004709 10490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:48.005862 10490 hybrid_clock.cc:648] HybridClock initialized: now 1781902248005850 us; error 26 us; skew 500 ppm
I20260619 20:50:48.007002 10490 webserver.cc:460] Webserver started at http://127.0.98.3:37649/ using document root <none> and password file <none>
I20260619 20:50:48.007272 10490 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:48.007340 10490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:48.007486 10490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:48.008477 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "21df39f7e16548409f8f9993c0b17d67"
format_stamp: "Formatted at 2026-06-19 20:50:48 on dist-test-slave-0smx"
server_key: "50a36dab582fa540a716df33a8964dda"
server_key_iv: "5fbec96177a85c5041740304a783e6e5"
server_key_version: "encryptionkey@0"
I20260619 20:50:48.008838 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21df39f7e16548409f8f9993c0b17d67"
format_stamp: "Formatted at 2026-06-19 20:50:48 on dist-test-slave-0smx"
server_key: "50a36dab582fa540a716df33a8964dda"
server_key_iv: "5fbec96177a85c5041740304a783e6e5"
server_key_version: "encryptionkey@0"
I20260619 20:50:48.010191 10490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:50:48.010910 10504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:48.011070 10490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:48.011163 10490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "21df39f7e16548409f8f9993c0b17d67"
format_stamp: "Formatted at 2026-06-19 20:50:48 on dist-test-slave-0smx"
server_key: "50a36dab582fa540a716df33a8964dda"
server_key_iv: "5fbec96177a85c5041740304a783e6e5"
server_key_version: "encryptionkey@0"
I20260619 20:50:48.011242 10490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:48.014479 10490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:48.014726 10490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:48.014885 10490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:48.015074 10490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:48.015358 10490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:48.015410 10490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:48.015465 10490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:48.015493 10490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:48.020949 10490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40045
I20260619 20:50:48.020983 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40045 every 8 connection(s)
I20260619 20:50:48.021284 10490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:50:48.022181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10490
I20260619 20:50:48.022274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:50:48.022441 392 external_mini_cluster.cc:1428] Setting key 7a89478172058f6a8d3cf51982bc67f0
I20260619 20:50:48.024287 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:48.022028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49049 (local address 127.0.98.62:44413)
0619 20:50:48.022099 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:48.022101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:48.022305 (+ 204us) server_negotiation.cc:407] Connection header received
0619 20:50:48.022857 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:48.022858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:48.022864 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:48.022931 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:48.024072 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:48.024073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:48.024074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:50:48.024077 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:48.024089 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:50:48.024092 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:48.024138 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:48.024225 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:50:48.024237 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:50:48.024451 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:48.021997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44413 (local address 127.0.98.3:49049)
0619 20:50:48.022194 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:48.022232 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:48.022751 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:48.022997 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:48.023001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:48.023010 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:48.023941 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:50:48.023941 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:48.023967 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:50:48.023992 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:48.024150 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:48.024153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:48.024156 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:50:48.024210 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:50:48.024253 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260619 20:50:48.024876 10618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44413
I20260619 20:50:48.024948 10618 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:48.025099 10618 heartbeater.cc:511] Master 127.0.98.62:44413 requested a full tablet report, sending...
I20260619 20:50:48.025460 10168 ts_manager.cc:194] Registered new tserver with Master: 21df39f7e16548409f8f9993c0b17d67 (127.0.98.3:40045)
I20260619 20:50:48.034310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:50:48.036208 10207 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:50:48.035700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.98.62:44413)
0619 20:50:48.035737 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:48.035739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:48.035869 (+ 130us) server_negotiation.cc:407] Connection header received
0619 20:50:48.035912 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:48.035913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:48.035917 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:48.035957 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:48.036130 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36764: BlockingRecv error: recv got EOF from 127.0.0.1:36764 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:50:48.036238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10228
I20260619 20:50:48.040557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10359
I20260619 20:50:48.044780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10490
I20260619 20:50:48.048952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10137
2026-06-19T20:50:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:50:48 dist-test-slave-0smx krb5kdc[10628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:50:48 dist-test-slave-0smx krb5kdc[10628](info): set up 2 sockets
Jun 19 20:50:48 dist-test-slave-0smx krb5kdc[10628](info): commencing operation
krb5kdc: starting...
W20260619 20:50:49.077652 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:50:49.187798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:49.265839 10641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:49.266034 10641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:49.266055 10641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:49.267138 10641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:50:49.267169 10641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:49.267182 10641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:49.267195 10641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:50:49.267207 10641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:50:49.268538 10641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:49.268774 10641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:49.268965 10641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:49.273569 10646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.273633 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.273594 10647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:49.273869 10641 server_base.cc:1034] running on GCE node
I20260619 20:50:49.274082 10641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:49.274308 10641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:49.275451 10641 hybrid_clock.cc:648] HybridClock initialized: now 1781902249275439 us; error 27 us; skew 500 ppm
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:49.277894 10641 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:50:49.278323 10641 webserver.cc:460] Webserver started at http://127.0.98.62:35943/ using document root <none> and password file <none>
I20260619 20:50:49.278543 10641 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:49.278589 10641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:49.278735 10641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:49.279783 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "497b37c9804043ea913f1a272e6e0866"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "0c13e9c6fc3dd34137c3912ac2eee5d6"
server_key_iv: "50634b57eeeede983ac98ba331b71b83"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.280130 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "497b37c9804043ea913f1a272e6e0866"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "0c13e9c6fc3dd34137c3912ac2eee5d6"
server_key_iv: "50634b57eeeede983ac98ba331b71b83"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.281571 10641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:50:49.282271 10656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.282457 10641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:49.282552 10641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "497b37c9804043ea913f1a272e6e0866"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "0c13e9c6fc3dd34137c3912ac2eee5d6"
server_key_iv: "50634b57eeeede983ac98ba331b71b83"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.282627 10641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:49.308179 10641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:49.310104 10641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:49.310271 10641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:49.314158 10641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41635
I20260619 20:50:49.314193 10708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41635 every 8 connection(s)
I20260619 20:50:49.314564 10641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:50:49.315531 10709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:49.317723 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866: Bootstrap starting.
I20260619 20:50:49.318218 10709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:49.318439 10709 log.cc:824] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:49.319054 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866: No bootstrap required, opened a new log
I20260619 20:50:49.320504 10709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497b37c9804043ea913f1a272e6e0866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260619 20:50:49.320631 10709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:49.320652 10709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497b37c9804043ea913f1a272e6e0866, State: Initialized, Role: FOLLOWER
I20260619 20:50:49.320756 10709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497b37c9804043ea913f1a272e6e0866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260619 20:50:49.320845 10709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:50:49.320883 10709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:50:49.320933 10709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:49.321730 10709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497b37c9804043ea913f1a272e6e0866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260619 20:50:49.321848 10709 leader_election.cc:302] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 497b37c9804043ea913f1a272e6e0866; no voters:
I20260619 20:50:49.322007 10709 leader_election.cc:288] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:50:49.322003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10641
I20260619 20:50:49.322095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:50:49.322079 10712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:49.322212 10712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [term 1 LEADER]: Becoming Leader. State: Replica: 497b37c9804043ea913f1a272e6e0866, State: Running, Role: LEADER
I20260619 20:50:49.322250 10709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:50:49.322304 392 external_mini_cluster.cc:1428] Setting key 2639c3ecd617f96b1de9bb00e8c4cffc
I20260619 20:50:49.322292 10712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497b37c9804043ea913f1a272e6e0866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260619 20:50:49.322701 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "497b37c9804043ea913f1a272e6e0866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497b37c9804043ea913f1a272e6e0866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260619 20:50:49.322789 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:49.323165 10714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 497b37c9804043ea913f1a272e6e0866. Latest consensus state: current_term: 1 leader_uuid: "497b37c9804043ea913f1a272e6e0866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497b37c9804043ea913f1a272e6e0866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260619 20:50:49.323292 10714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866 [sys.catalog]: This master's current role is: LEADER
I20260619 20:50:49.323182 10721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:50:49.323917 10721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:50:49.325481 10721 catalog_manager.cc:1347] Generated new cluster ID: 2176f8fee41b470489e79a354b6954e2
I20260619 20:50:49.325531 10721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:49.332535 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.322893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.0.98.62:41635)
0619 20:50:49.323067 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.323076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.323341 (+ 265us) server_negotiation.cc:407] Connection header received
0619 20:50:49.323449 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.323456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.323535 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.323670 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:49.324546 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.325810 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.326578 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.326985 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.327113 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.329412 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:49.329415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:49.329415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:49.329425 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:49.331041 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.331415 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.331418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.331418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.331511 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.331720 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.331721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.331722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.331937 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:49.332101 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.332195 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.332332 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260619 20:50:49.341346 10721 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:50:49.342020 10721 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:50:49.347538 10721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866: Generated new TSK 0
I20260619 20:50:49.347693 10721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:50:49.380301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:49.459331 10735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:49.459548 10735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:49.459583 10735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:49.460702 10735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:49.460772 10735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:49.460817 10735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:50:49.462265 10735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:49.462527 10735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:49.462754 10735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:49.467659 10741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:49.467710 10735 server_base.cc:1034] running on GCE node
W20260619 20:50:49.467656 10743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.467718 10740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:49.468080 10735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:49.468287 10735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:49.469447 10735 hybrid_clock.cc:648] HybridClock initialized: now 1781902249469431 us; error 28 us; skew 500 ppm
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:49.471812 10735 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:50:49.472218 10735 webserver.cc:460] Webserver started at http://127.0.98.1:36673/ using document root <none> and password file <none>
I20260619 20:50:49.472452 10735 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:49.472520 10735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:49.472675 10735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:49.473753 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "af3f615edd2e41e0994334a42bc858b0"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "68ce509f502dd2b6c56a27d193f8432e"
server_key_iv: "a0a96de16619d479a51d7d826338814d"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.474115 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "af3f615edd2e41e0994334a42bc858b0"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "68ce509f502dd2b6c56a27d193f8432e"
server_key_iv: "a0a96de16619d479a51d7d826338814d"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.475455 10735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:49.476140 10750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.476306 10735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:49.476406 10735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "af3f615edd2e41e0994334a42bc858b0"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "68ce509f502dd2b6c56a27d193f8432e"
server_key_iv: "a0a96de16619d479a51d7d826338814d"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.476485 10735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:49.497605 10735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:49.499400 10735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:49.499562 10735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:49.499794 10735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:49.500106 10735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:49.500164 10735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.500198 10735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:49.500226 10735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.505795 10735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44241
I20260619 20:50:49.505869 10863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44241 every 8 connection(s)
I20260619 20:50:49.506186 10735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:49.514587 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.506886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47763 (local address 127.0.98.62:41635)
0619 20:50:49.506929 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.506931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.507132 (+ 201us) server_negotiation.cc:407] Connection header received
0619 20:50:49.507588 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.507589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.507601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.507656 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:49.508545 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.509020 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.510033 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.510229 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.510282 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.512207 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:49.512208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:49.512208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:49.512215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:49.513436 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.513845 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.513846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.513846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.513893 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.514100 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.514101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.514101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.514201 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:49.514248 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.514465 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.514527 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:50:49.514806 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.506841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:47763)
0619 20:50:49.507000 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:49.507041 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:49.507506 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:49.507728 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:49.507732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:49.508292 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:49.508469 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.508474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.509104 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.509104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.509957 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.509960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.510301 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.510302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.510400 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.510826 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:49.510833 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:49.512108 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:49.513502 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:49.513508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:49.513509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:49.513782 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:49.513948 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:49.513949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:49.513949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:49.514049 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:49.514253 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:49.514257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:49.514360 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 20:50:49.514439 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:50:49.514546 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260619 20:50:49.515153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10735
I20260619 20:50:49.515249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:50:49.515338 10864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260619 20:50:49.515425 10864 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:49.515466 392 external_mini_cluster.cc:1428] Setting key 42e47ab57a07f89cef400dfbb9d26904
I20260619 20:50:49.515609 10864 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260619 20:50:49.516112 10672 ts_manager.cc:194] Registered new tserver with Master: af3f615edd2e41e0994334a42bc858b0 (127.0.98.1:44241)
I20260619 20:50:49.516726 10672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47763
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:50:49.540119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:49.619699 10869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:49.619896 10869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:49.619920 10869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:49.620997 10869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:49.621031 10869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:49.621059 10869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:50:49.622592 10869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:49.622763 10869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:49.622931 10869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:49.627741 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.627746 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.627751 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:49.627938 10869 server_base.cc:1034] running on GCE node
I20260619 20:50:49.628187 10869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:49.628371 10869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:49.629542 10869 hybrid_clock.cc:648] HybridClock initialized: now 1781902249629534 us; error 38 us; skew 500 ppm
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:49.631886 10869 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:50:49.632266 10869 webserver.cc:460] Webserver started at http://127.0.98.2:40155/ using document root <none> and password file <none>
I20260619 20:50:49.632479 10869 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:49.632546 10869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:49.632689 10869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:49.633776 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "18ccfbb3e34b4d69b78eca909bc60fdd"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "cd2ecb93dd8feba7446504d7cf147c1e"
server_key_iv: "019a9654e2a8248e2d6c0baaa849a3a0"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.634119 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "18ccfbb3e34b4d69b78eca909bc60fdd"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "cd2ecb93dd8feba7446504d7cf147c1e"
server_key_iv: "019a9654e2a8248e2d6c0baaa849a3a0"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.635365 10869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:49.636106 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.636277 10869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:49.636373 10869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "18ccfbb3e34b4d69b78eca909bc60fdd"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "cd2ecb93dd8feba7446504d7cf147c1e"
server_key_iv: "019a9654e2a8248e2d6c0baaa849a3a0"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.636449 10869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:49.645695 10869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:49.647492 10869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:49.647648 10869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:49.647863 10869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:49.648161 10869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:49.648222 10869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.648257 10869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:49.648283 10869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.653947 10869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42251
I20260619 20:50:49.654054 10997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42251 every 8 connection(s)
I20260619 20:50:49.654328 10869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:50:49.654829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 10869
I20260619 20:50:49.654942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:50:49.655170 392 external_mini_cluster.cc:1428] Setting key e704e1b9f7a5c18d6e4f2efde53e5634
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:50:49.664292 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.655011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56715 (local address 127.0.98.62:41635)
0619 20:50:49.655061 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.655063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.655277 (+ 214us) server_negotiation.cc:407] Connection header received
0619 20:50:49.655974 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.655975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.655986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.656036 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:49.656960 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.657560 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.658546 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.658753 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.658836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.661227 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:49.661228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:49.661229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:49.661236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:49.662765 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.663211 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.663213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.663213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.663268 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.663527 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.663528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.663529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.663652 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:49.663731 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.664096 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.664166 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:50:49.664496 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.655003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:56715)
0619 20:50:49.655162 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:49.655201 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:49.655871 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:49.656108 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:49.656112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:49.656674 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:49.656859 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.656863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.657649 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.657650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.658441 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.658444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.658839 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.658840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.658985 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.659511 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:49.659519 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:49.661101 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:49.662877 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:49.662884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:49.662886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:49.663130 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:49.663351 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:49.663351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:49.663352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:49.663454 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:49.663721 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:49.663727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:49.663939 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 20:50:49.664058 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:50:49.664219 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260619 20:50:49.665262 10998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260619 20:50:49.665377 10998 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:49.665560 10998 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 20:50:49.666620 10672 ts_manager.cc:194] Registered new tserver with Master: 18ccfbb3e34b4d69b78eca909bc60fdd (127.0.98.2:42251)
I20260619 20:50:49.667106 10672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:50:49.680676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:50:49.760340 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:50:49.760533 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:50:49.760555 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:50:49.761633 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:50:49.761673 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:50:49.761699 11003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:50:49.763015 11003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:50:49.763180 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:50:49.763343 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:50:49.768285 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.768369 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:50:49.768281 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:50:49.768298 11003 server_base.cc:1034] running on GCE node
I20260619 20:50:49.768687 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:50:49.768863 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:50:49.769987 11003 hybrid_clock.cc:648] HybridClock initialized: now 1781902249769966 us; error 34 us; skew 500 ppm
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:50:49.772311 11003 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:50:49.772704 11003 webserver.cc:460] Webserver started at http://127.0.98.3:39095/ using document root <none> and password file <none>
I20260619 20:50:49.772883 11003 fs_manager.cc:359] Metadata directory not provided
I20260619 20:50:49.772922 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:50:49.773005 11003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:50:49.774053 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "f3c42ed0bf9a8dff6610a55a3f235f3d"
server_key_iv: "e609987046fcfcc04224f537152ad1c0"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.774343 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "f3c42ed0bf9a8dff6610a55a3f235f3d"
server_key_iv: "e609987046fcfcc04224f537152ad1c0"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.775484 11003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:50:49.776197 11018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.776386 11003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:50:49.776448 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a"
format_stamp: "Formatted at 2026-06-19 20:50:49 on dist-test-slave-0smx"
server_key: "f3c42ed0bf9a8dff6610a55a3f235f3d"
server_key_iv: "e609987046fcfcc04224f537152ad1c0"
server_key_version: "encryptionkey@0"
I20260619 20:50:49.776530 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:50:49.796119 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:50:49.798221 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:50:49.798377 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:50:49.798638 11003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:50:49.798940 11003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:50:49.798972 11003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.799018 11003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:50:49.799047 11003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:50:49.805171 11003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44561
I20260619 20:50:49.805578 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:50:49.807401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 11003
I20260619 20:50:49.807511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:50:49.807767 392 external_mini_cluster.cc:1428] Setting key d9ee04fa95b0a7d54c3a8f7015097517
I20260619 20:50:49.808928 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44561 every 8 connection(s)
Jun 19 20:50:49 dist-test-slave-0smx krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:50:49.817255 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.809541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46015 (local address 127.0.98.62:41635)
0619 20:50:49.809585 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.809587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.809861 (+ 274us) server_negotiation.cc:407] Connection header received
0619 20:50:49.810295 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.810296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.810305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.810354 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:49.811327 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.811810 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.812747 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.812920 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.812968 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.815018 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:49.815020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:49.815020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:49.815026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:49.816189 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.816526 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.816527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.816527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.816574 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.816776 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.816777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.816777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.816869 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:49.816918 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.817137 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.817194 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:50:49.817485 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.809554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:46015)
0619 20:50:49.809737 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:49.809775 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:49.810224 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:49.810406 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:49.810409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:49.810974 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:50:49.811250 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.811256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.811885 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.811885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.812677 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.812679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.812988 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.812989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.813083 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.813642 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:50:49.813650 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:50:49.814938 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:50:49.816259 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:49.816264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:49.816266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:49.816464 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:49.816629 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:50:49.816630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:50:49.816630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:50:49.816725 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:50:49.816923 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:50:49.816926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:50:49.817030 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 20:50:49.817109 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:50:49.817220 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260619 20:50:49.817993 11132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260619 20:50:49.818110 11132 heartbeater.cc:464] Registering TS with master...
I20260619 20:50:49.818280 11132 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260619 20:50:49.818686 10672 ts_manager.cc:194] Registered new tserver with Master: 9ed756ed49d6442b9ffbd7fb3f4eea6a (127.0.98.3:44561)
I20260619 20:50:49.819074 10672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46015
I20260619 20:50:49.819667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:50:49.827049 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.821005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.0.98.62:41635)
0619 20:50:49.821063 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.821065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.821177 (+ 112us) server_negotiation.cc:407] Connection header received
0619 20:50:49.821262 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.821263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.821273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.821345 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:50:49.821995 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.822454 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.823352 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.823535 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.823611 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.824822 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:50:49.824823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:50:49.824823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:50:49.824830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:50:49.825953 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.826295 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.826296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.826297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.826348 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:50:49.826545 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:50:49.826545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:50:49.826546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:50:49.826675 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:50:49.826780 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.826883 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.826954 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:50:49.828608 10673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:50:49.830215 10673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:50:49.836236 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.832576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.0.98.1:44241)
0619 20:50:49.833007 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.833012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.833022 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:50:49.833059 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.833061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.833076 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.833145 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:49.833388 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.834208 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.835229 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.835818 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.835891 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.835948 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.836112 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.836152 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":81,"threads_started":1}
I20260619 20:50:49.836256 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.832414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44241 (local address 127.0.0.1:60266)
0619 20:50:49.832766 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:49.832810 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:49.832877 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:49.833228 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:49.833229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:49.833231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:49.833309 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.833311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.834312 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.834313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.835107 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.835109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.835903 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.835903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.835983 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.835997 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:50:49.836091 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:50:49.836149 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":5,"thread_start_us":70,"threads_started":1}
I20260619 20:50:49.837129 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.832429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44561 (local address 127.0.0.1:50758)
0619 20:50:49.832767 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:49.832809 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:49.832877 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:49.833123 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:49.833126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:49.833128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:49.833199 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.833202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.834561 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.834562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.835668 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.835670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.836861 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.836862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.836935 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.836939 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:49.837021 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:50:49.837060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":78,"threads_started":1}
I20260619 20:50:49.837041 10798 tablet_service.cc:1467] Processing CreateTablet for tablet 48f579b375634a999ad0d3425738c985 (DEFAULT_TABLE table=test-table [id=00d81ddae5e24e2687725c905e030f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:49.837146 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.832510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.0.98.3:44561)
0619 20:50:49.832833 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.832838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.832854 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:50:49.832996 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.833000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.833016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.833101 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:49.833329 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.834429 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.835763 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.836753 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.836857 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.836910 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.837037 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.837073 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":79,"threads_started":1}
I20260619 20:50:49.837445 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.832668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42251 (local address 127.0.0.1:51170)
0619 20:50:49.833201 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:49.833211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:49.833259 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:49.833563 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:49.833564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:49.833566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:49.833598 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.833599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.834864 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.834865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.836157 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:49.836159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.837156 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.837157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:49.837218 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.837223 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:50:49.837282 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:50:49.837361 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":49,"threads_started":1}
I20260619 20:50:49.837446 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f579b375634a999ad0d3425738c985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:49.837671 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:49.832805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.0.98.2:42251)
0619 20:50:49.832967 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:49.832970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:49.833340 (+ 370us) server_negotiation.cc:407] Connection header received
0619 20:50:49.833385 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:49.833388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:49.833402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:49.833456 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:49.833683 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.834744 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.836252 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:49.836981 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:49.837062 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:49.837116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:49.837548 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 20:50:49.837592 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260619 20:50:49.838294 11048 tablet_service.cc:1467] Processing CreateTablet for tablet 48f579b375634a999ad0d3425738c985 (DEFAULT_TABLE table=test-table [id=00d81ddae5e24e2687725c905e030f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:49.838650 11048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f579b375634a999ad0d3425738c985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:49.841046 11146 tablet_bootstrap.cc:492] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: Bootstrap starting.
I20260619 20:50:49.841498 10932 tablet_service.cc:1467] Processing CreateTablet for tablet 48f579b375634a999ad0d3425738c985 (DEFAULT_TABLE table=test-table [id=00d81ddae5e24e2687725c905e030f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:50:49.841732 11146 tablet_bootstrap.cc:654] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:49.841846 10932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f579b375634a999ad0d3425738c985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:50:49.842068 11147 tablet_bootstrap.cc:492] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Bootstrap starting.
I20260619 20:50:49.842381 11146 log.cc:824] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:49.842772 11147 tablet_bootstrap.cc:654] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:49.843310 11146 tablet_bootstrap.cc:492] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: No bootstrap required, opened a new log
I20260619 20:50:49.843371 11147 log.cc:824] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:49.843389 11146 ts_tablet_manager.cc:1397] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:50:49.844267 11147 tablet_bootstrap.cc:492] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: No bootstrap required, opened a new log
I20260619 20:50:49.844336 11147 ts_tablet_manager.cc:1397] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:50:49.844785 11150 tablet_bootstrap.cc:492] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Bootstrap starting.
I20260619 20:50:49.845433 11150 tablet_bootstrap.cc:654] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Neither blocks nor log segments found. Creating new log.
I20260619 20:50:49.845607 11146 raft_consensus.cc:348] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:49.845803 11146 raft_consensus.cc:374] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:49.845839 11146 raft_consensus.cc:729] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3f615edd2e41e0994334a42bc858b0, State: Initialized, Role: FOLLOWER
I20260619 20:50:49.845966 11150 log.cc:824] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Log is configured to *not* fsync() on all Append() calls
I20260619 20:50:49.845973 11146 consensus_queue.cc:260] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:49.846226 11146 ts_tablet_manager.cc:1428] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:50:49.846349 10864 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260619 20:50:49.846266 11147 raft_consensus.cc:348] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:49.846647 11147 raft_consensus.cc:374] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:49.846701 11147 raft_consensus.cc:729] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed756ed49d6442b9ffbd7fb3f4eea6a, State: Initialized, Role: FOLLOWER
I20260619 20:50:49.846835 11150 tablet_bootstrap.cc:492] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: No bootstrap required, opened a new log
I20260619 20:50:49.846845 11147 consensus_queue.cc:260] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:49.846930 11150 ts_tablet_manager.cc:1397] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:50:49.847091 11147 ts_tablet_manager.cc:1428] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:50:49.847249 11132 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260619 20:50:49.848812 11150 raft_consensus.cc:348] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:49.848968 11150 raft_consensus.cc:374] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:50:49.849020 11150 raft_consensus.cc:729] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ccfbb3e34b4d69b78eca909bc60fdd, State: Initialized, Role: FOLLOWER
I20260619 20:50:49.849134 11150 consensus_queue.cc:260] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:49.849442 11150 ts_tablet_manager.cc:1428] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:50:49.849452 10998 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
W20260619 20:50:49.904950 10999 tablet.cc:2369] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:50:50.006937 10865 tablet.cc:2369] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:50:50.058473 11133 tablet.cc:2369] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:50:50.111899 11151 raft_consensus.cc:482] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:50:50.112020 11151 raft_consensus.cc:504] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:50.112378 11151 leader_election.cc:288] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed756ed49d6442b9ffbd7fb3f4eea6a (127.0.98.3:44561), 18ccfbb3e34b4d69b78eca909bc60fdd (127.0.98.2:42251)
I20260619 20:50:50.115614 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:50.112560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44561 (local address 127.0.98.1:47377)
0619 20:50:50.112725 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:50.112736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:50.112818 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:50.112961 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:50.112962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:50.112964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:50.113050 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:50.113052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.113916 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.113917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:50.114776 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:50.114778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.115342 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.115343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:50.115414 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:50.115418 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:50.115478 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:50:50.115520 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":29,"thread_start_us":76,"threads_started":1}
I20260619 20:50:50.115602 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:50.112651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47377 (local address 127.0.98.3:44561)
0619 20:50:50.112702 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:50.112704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:50.112806 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:50:50.112877 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:50.112878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:50.112889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:50.112959 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:50.113127 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.113841 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.114869 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.115248 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.115328 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:50.115339 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:50.115495 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:50:50.115537 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:50:50.115620 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:50.112672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51753 (local address 127.0.98.2:42251)
0619 20:50:50.112734 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:50.112736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:50.112821 (+ 85us) server_negotiation.cc:407] Connection header received
0619 20:50:50.112873 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:50.112874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:50.112888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:50.112963 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:50:50.113164 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.113829 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.114875 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.115259 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.115354 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:50.115363 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:50.115519 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:50:50.115556 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:50:50.115633 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:50.112600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42251 (local address 127.0.98.1:51753)
0619 20:50:50.112736 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:50.112747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:50:50.112813 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:50:50.112967 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:50:50.112967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:50:50.112969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:50:50.113075 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:50.113077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.113907 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.113908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:50.114778 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:50:50.114780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.115356 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.115356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:50:50.115427 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:50.115431 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:50:50.115502 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:50:50.115543 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":66,"thread_start_us":60,"threads_started":1}
I20260619 20:50:50.115949 11086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f579b375634a999ad0d3425738c985" candidate_uuid: "af3f615edd2e41e0994334a42bc858b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" is_pre_election: true
I20260619 20:50:50.116009 10952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f579b375634a999ad0d3425738c985" candidate_uuid: "af3f615edd2e41e0994334a42bc858b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" is_pre_election: true
I20260619 20:50:50.116112 11086 raft_consensus.cc:2454] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3f615edd2e41e0994334a42bc858b0 in term 0.
I20260619 20:50:50.116112 10952 raft_consensus.cc:2454] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3f615edd2e41e0994334a42bc858b0 in term 0.
I20260619 20:50:50.116359 10752 leader_election.cc:302] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ccfbb3e34b4d69b78eca909bc60fdd, af3f615edd2e41e0994334a42bc858b0; no voters:
I20260619 20:50:50.116515 11151 raft_consensus.cc:2781] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:50:50.116559 11151 raft_consensus.cc:482] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:50:50.116575 11151 raft_consensus.cc:3037] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:50.117230 11151 raft_consensus.cc:504] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:50.117399 11151 leader_election.cc:288] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [CANDIDATE]: Term 1 election: Requested vote from peers 9ed756ed49d6442b9ffbd7fb3f4eea6a (127.0.98.3:44561), 18ccfbb3e34b4d69b78eca909bc60fdd (127.0.98.2:42251)
I20260619 20:50:50.117524 11086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f579b375634a999ad0d3425738c985" candidate_uuid: "af3f615edd2e41e0994334a42bc858b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a"
I20260619 20:50:50.117550 10952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f579b375634a999ad0d3425738c985" candidate_uuid: "af3f615edd2e41e0994334a42bc858b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd"
I20260619 20:50:50.117597 11086 raft_consensus.cc:3037] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:50.117635 10952 raft_consensus.cc:3037] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:50:50.118261 11086 raft_consensus.cc:2454] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3f615edd2e41e0994334a42bc858b0 in term 1.
I20260619 20:50:50.118403 10952 raft_consensus.cc:2454] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3f615edd2e41e0994334a42bc858b0 in term 1.
I20260619 20:50:50.118438 10754 leader_election.cc:302] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed756ed49d6442b9ffbd7fb3f4eea6a, af3f615edd2e41e0994334a42bc858b0; no voters:
I20260619 20:50:50.118517 11151 raft_consensus.cc:2781] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:50:50.118647 11151 raft_consensus.cc:686] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [term 1 LEADER]: Becoming Leader. State: Replica: af3f615edd2e41e0994334a42bc858b0, State: Running, Role: LEADER
I20260619 20:50:50.118741 11151 consensus_queue.cc:237] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } }
I20260619 20:50:50.119519 10672 catalog_manager.cc:5495] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3f615edd2e41e0994334a42bc858b0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "af3f615edd2e41e0994334a42bc858b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3f615edd2e41e0994334a42bc858b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 } health_report { overall_health: UNKNOWN } } }
I20260619 20:50:50.188233 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:50.185880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.0.98.1:44241)
0619 20:50:50.185949 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:50:50.185951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:50:50.185964 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:50:50.186063 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:50:50.186063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:50:50.186084 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:50:50.186150 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:50:50.186285 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.186775 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.187503 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:50:50.187688 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:50:50.187736 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:50:50.187881 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:50:50.187981 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:50:50.188044 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:50:50.188156 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 20:50:50.188174 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:50:50.190155 11086 raft_consensus.cc:1261] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 1 FOLLOWER]: Refusing update from remote peer af3f615edd2e41e0994334a42bc858b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:50.190165 10952 raft_consensus.cc:1261] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 1 FOLLOWER]: Refusing update from remote peer af3f615edd2e41e0994334a42bc858b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:50:50.190518 11157 consensus_queue.cc:1035] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ccfbb3e34b4d69b78eca909bc60fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:50.190642 11151 consensus_queue.cc:1035] T 48f579b375634a999ad0d3425738c985 P af3f615edd2e41e0994334a42bc858b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed756ed49d6442b9ffbd7fb3f4eea6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:50:50.192749 11160 mvcc.cc:205] Tried to move back new op lower bound from 7298671616776421376 to 7298671616486965248. Current Snapshot: MvccSnapshot[applied={T|T < 7298671616776421376}]
I20260619 20:50:50.193771 11161 mvcc.cc:205] Tried to move back new op lower bound from 7298671616776421376 to 7298671616486965248. Current Snapshot: MvccSnapshot[applied={T|T < 7298671616776421376 or (T in {7298671616776421376})}]
I20260619 20:50:50.195405 10673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:50.195515 10673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:50:50.196226 10673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 497b37c9804043ea913f1a272e6e0866: Sending DeleteTablet for 3 replicas of tablet 48f579b375634a999ad0d3425738c985
I20260619 20:50:50.196554 11048 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f579b375634a999ad0d3425738c985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50758
I20260619 20:50:50.196575 10798 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f579b375634a999ad0d3425738c985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60266
I20260619 20:50:50.196578 10932 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f579b375634a999ad0d3425738c985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:50:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51170
I20260619 20:50:50.196805 11171 tablet_replica.cc:330] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: stopping tablet replica
I20260619 20:50:50.196898 11171 raft_consensus.cc:2229] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:50.197031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10735
I20260619 20:50:50.197796 11171 raft_consensus.cc:2258] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:50.197908 11169 tablet_replica.cc:330] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: stopping tablet replica
I20260619 20:50:50.198055 11169 raft_consensus.cc:2229] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:50:50.198196 11169 raft_consensus.cc:2258] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:50:50.198253 11171 ts_tablet_manager.cc:1905] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:50.198627 11169 ts_tablet_manager.cc:1905] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:50:50.200222 11171 ts_tablet_manager.cc:1918] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:50.200296 11171 log.cc:1196] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/48f579b375634a999ad0d3425738c985
I20260619 20:50:50.200508 11169 ts_tablet_manager.cc:1918] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:50:50.200555 11171 ts_tablet_manager.cc:1939] T 48f579b375634a999ad0d3425738c985 P 18ccfbb3e34b4d69b78eca909bc60fdd: Deleting consensus metadata
I20260619 20:50:50.200579 11169 log.cc:1196] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/48f579b375634a999ad0d3425738c985
I20260619 20:50:50.200836 11169 ts_tablet_manager.cc:1939] T 48f579b375634a999ad0d3425738c985 P 9ed756ed49d6442b9ffbd7fb3f4eea6a: Deleting consensus metadata
I20260619 20:50:50.200944 10658 catalog_manager.cc:4841] TS 18ccfbb3e34b4d69b78eca909bc60fdd (127.0.98.2:42251): tablet 48f579b375634a999ad0d3425738c985 (table test-table [id=00d81ddae5e24e2687725c905e030f65]) successfully deleted
I20260619 20:50:50.201215 10660 catalog_manager.cc:4841] TS 9ed756ed49d6442b9ffbd7fb3f4eea6a (127.0.98.3:44561): tablet 48f579b375634a999ad0d3425738c985 (table test-table [id=00d81ddae5e24e2687725c905e030f65]) successfully deleted
W20260619 20:50:50.202692 10658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44241 (error 108)
I20260619 20:50:50.202875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10869
I20260619 20:50:50.203512 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:50:50.203033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44241 (local address 127.0.0.1:60282)
0619 20:50:50.203457 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 20:50:50.203489 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401}
W20260619 20:50:50.203722 10658 catalog_manager.cc:4584] TS af3f615edd2e41e0994334a42bc858b0 (127.0.98.1:44241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48f579b375634a999ad0d3425738c985: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44241: connect: Connection refused (error 111)
I20260619 20:50:50.207387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 11003
I20260619 20:50:50.211789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 10641
2026-06-19T20:50:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2162 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:50:50 dist-test-slave-0smx krb5kdc[11175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:50:50 dist-test-slave-0smx krb5kdc[11175](info): set up 2 sockets
Jun 19 20:50:50 dist-test-slave-0smx krb5kdc[11175](info): commencing operation
krb5kdc: starting...
W20260619 20:50:51.249716 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:50:51 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:50:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 20:50:51.411798 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskH2GnNK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/postgres -l logfile start
2026-06-19 20:50:53.817 UTC [11205] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 20:50:53.817 UTC [11205] LOG: listening on IPv4 address "127.0.98.20", port 52953
2026-06-19 20:50:53.820 UTC [11205] LOG: listening on Unix socket "/tmp/.s.PGSQL.52953"
2026-06-19 20:50:53.854 UTC [11208] LOG: database system was shut down at 2026-06-19 20:50:52 UTC
2026-06-19 20:50:53.858 UTC [11205] LOG: database system is ready to accept connections
I20260619 20:50:54.797456 392 mini_postgres.cc:96] Postgres bound to 52953
2026-06-19 20:50:54.803 UTC [11216] FATAL: database "slave" does not exist
127.0.98.20:52953 - accepting connections
I20260619 20:50:54.803982 392 mini_ranger.cc:161] Starting Ranger...
I20260619 20:50:54.818257 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 20:50:55.286904 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 20:50:55.286985 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-admin
2026-06-19 20:50:55,567 [I] DB FLAVOR :POSTGRES
2026-06-19 20:50:55,567 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:50:55,567 [I] Checking connection..
2026-06-19 20:50:55,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 20:50:55,811 [I] Checking connection passed.
2026-06-19 20:50:55,811 [I] --------- Verifying version history table ---------
2026-06-19 20:50:55,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:50:56,047 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:50:56,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:50:56,305 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:50:56,306 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 20:50:56,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 20:50:56.560 UTC [11280] WARNING: there is no transaction in progress
2026-06-19 20:50:56,566 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 20:50:56,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:50:56,807 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 20:50:56,807 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:50:56,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 20:50:57,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 20:50:57,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0smx.c.gcp-upstream.internal','N') ;"
2026-06-19 20:50:57,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:50:57,768 [I] Table x_portal_user does not exist in database ranger
2026-06-19 20:50:57,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:50:58,023 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 20:50:58,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:50:58,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:50:58,509 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 20:50:58,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 20:50:59.408 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.422 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.438 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.452 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.469 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.488 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.547 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.553 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.566 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.576 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.585 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.598 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.607 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.613 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.619 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:50:59.625 UTC [11388] WARNING: there is no transaction in progress
2026-06-19 20:51:00,216 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 20:51:00,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 20:51:00,466 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 20:51:00,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:51:00,696 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 20:51:00,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:51:00,922 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 20:51:00,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:51:01,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:51:01,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0smx.c.gcp-upstream.internal';"
2026-06-19 20:51:01,658 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 20:51:01,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 20:51:01,898 [I] DB_PATCHES have already been applied
I20260619 20:51:01.904327 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskH2GnNK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 20:51:01.904426 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 20:51:01.907550 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40467
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:51:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 20:51:03.020889 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.217s user 0.002s sys 0.007s
Jun 19, 2026 8:51:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:51:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 8:51:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:51:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:51:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 20:51:24.255580 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 20:51:24 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902284, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 20:51:24.332731 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 20:51:24.342093 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 20:51:25.147488 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 20:51:25.147603 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms
I20260619 20:51:25.147624 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52953
2026-06-19 20:51:25,315 [I] DB FLAVOR :POSTGRES
2026-06-19 20:51:25,316 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:51:25,316 [I] Checking connection
2026-06-19 20:51:25,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 20:51:25,560 [I] connection success
2026-06-19 20:51:25,560 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 20:51:25,560 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 20:51:25,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 20:51:25,821 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 20:51:25,821 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:51:25,822 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 20:51:25,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 20:51:26,094 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 20:51:26.560415 392 mini_ranger_kms.cc:310] Created kms service
I20260619 20:51:26.769019 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 20:51:26.809475 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 20:51:26 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902284, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:51:27.162395 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 20:51:27.162510 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 20:51:27.162532 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 20:51:27.166008 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11568
I20260619 20:51:27.166112 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:41155
Jun 19, 2026 8:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 20:51:29.449517 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.117s user 0.009s sys 0.141s
I20260619 20:51:29.449627 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 20:51:29.449735 392 mini_ranger_kms.cc:401] 127.0.98.20:41155/kms/v1/keys
Jun 19 20:51:29 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902289, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 20:51:30 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902289, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:51:30 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902289, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:51:32 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:51:32.514266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36285
--ranger_config_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41155/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:51:32.616873 11619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:51:32.617094 11619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:51:32.617117 11619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:51:32.618325 11619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:51:32.618376 11619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:51:32.618393 11619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:51:32.618446 11619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:51:32.618474 11619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:51:32.620093 11619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:41155/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36285
--ranger_config_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:51:32.620344 11619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:51:32.620570 11619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:51:32.625768 11624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:32.625833 11619 server_base.cc:1034] running on GCE node
W20260619 20:51:32.625860 11627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:51:32.625958 11625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:32.626300 11619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:51:32.626596 11619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:51:32.627746 11619 hybrid_clock.cc:648] HybridClock initialized: now 1781902292627746 us; error 40 us; skew 500 ppm
Jun 19 20:51:32 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:51:32.633440 11619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:51:32.633888 11619 webserver.cc:460] Webserver started at http://127.0.98.62:43469/ using document root <none> and password file <none>
I20260619 20:51:32.634105 11619 fs_manager.cc:359] Metadata directory not provided
I20260619 20:51:32.634159 11619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:51:32.634295 11619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:51:32 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:51:32.680850 11619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "c5bc8f40f2644cb1895443fbc634bc30"
format_stamp: "Formatted at 2026-06-19 20:51:32 on dist-test-slave-0smx"
server_key: "30ca9e5b65f827fb15cad10e6382d3f9"
server_key_iv: "596e66a030ea827e052138c9b6f5b69f"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:32.681329 11619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5bc8f40f2644cb1895443fbc634bc30"
format_stamp: "Formatted at 2026-06-19 20:51:32 on dist-test-slave-0smx"
server_key: "30ca9e5b65f827fb15cad10e6382d3f9"
server_key_iv: "596e66a030ea827e052138c9b6f5b69f"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:32.682751 11619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:51:32.697939 11634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:32.698203 11619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:51:32.698319 11619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "c5bc8f40f2644cb1895443fbc634bc30"
format_stamp: "Formatted at 2026-06-19 20:51:32 on dist-test-slave-0smx"
server_key: "30ca9e5b65f827fb15cad10e6382d3f9"
server_key_iv: "596e66a030ea827e052138c9b6f5b69f"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:32.698418 11619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:51:32.711175 11619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:51:32.713093 11619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:51:32.713236 11619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:51:32.717811 11619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36285
I20260619 20:51:32.717870 11686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36285 every 8 connection(s)
I20260619 20:51:32.718189 11619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:51:32.718791 11687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:51:32.720396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 11619
I20260619 20:51:32.720474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:51:32.720661 392 external_mini_cluster.cc:1428] Setting key 1ae0b4714fd20dd13fe0fb2449a8f9d3
I20260619 20:51:32.721253 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30: Bootstrap starting.
I20260619 20:51:32.721903 11687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30: Neither blocks nor log segments found. Creating new log.
I20260619 20:51:32.722189 11687 log.cc:824] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30: Log is configured to *not* fsync() on all Append() calls
I20260619 20:51:32.722824 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30: No bootstrap required, opened a new log
I20260619 20:51:32.724577 11687 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bc8f40f2644cb1895443fbc634bc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260619 20:51:32.724727 11687 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:51:32.724786 11687 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bc8f40f2644cb1895443fbc634bc30, State: Initialized, Role: FOLLOWER
I20260619 20:51:32.724985 11687 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bc8f40f2644cb1895443fbc634bc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260619 20:51:32.725091 11687 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:51:32.725129 11687 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:51:32.725191 11687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:51:32.725867 11687 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bc8f40f2644cb1895443fbc634bc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260619 20:51:32.725950 11687 leader_election.cc:302] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5bc8f40f2644cb1895443fbc634bc30; no voters:
I20260619 20:51:32.726135 11687 leader_election.cc:288] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:51:32.726161 11692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:51:32.726269 11692 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [term 1 LEADER]: Becoming Leader. State: Replica: c5bc8f40f2644cb1895443fbc634bc30, State: Running, Role: LEADER
I20260619 20:51:32.726399 11687 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 20:51:32 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:51:32.726418 11692 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bc8f40f2644cb1895443fbc634bc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260619 20:51:32.728621 11687 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:51:32.728629 11693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bc8f40f2644cb1895443fbc634bc30. Latest consensus state: current_term: 1 leader_uuid: "c5bc8f40f2644cb1895443fbc634bc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bc8f40f2644cb1895443fbc634bc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260619 20:51:32.728736 11693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [sys.catalog]: This master's current role is: LEADER
I20260619 20:51:32.728751 11694 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5bc8f40f2644cb1895443fbc634bc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bc8f40f2644cb1895443fbc634bc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260619 20:51:32.728814 11694 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30 [sys.catalog]: This master's current role is: LEADER
I20260619 20:51:32.731696 11706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:51:32.732700 11706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:51:32.734273 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:32.721144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.98.62:36285)
0619 20:51:32.721323 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:32.721332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:32.721362 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:51:32.721823 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:32.721831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:32.721878 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:32.722025 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:51:32.722700 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:32.723596 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:32.724356 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:32.724783 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:32.724902 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:32.727671 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:51:32.727675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:51:32.727675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:51:32.727687 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:51:32.731447 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:32.731996 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:32.732013 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:32.732014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:32.732231 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:32.732454 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:32.732454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:32.732455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:32.732963 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:51:32.733243 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:32.733342 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:51:32.733857 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260619 20:51:32.740232 11706 catalog_manager.cc:1347] Generated new cluster ID: 5d921320e68e46bf9ab4da63a4f0f67f
I20260619 20:51:32.740293 11706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:51:32.745949 11706 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:51:32.746647 11706 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:51:32.753531 11706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30: Generated new TSK 0
I20260619 20:51:32.753702 11706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:51:33.417979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:51:33.510056 11732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:51:33.510286 11732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:51:33.510326 11732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:51:33.511507 11732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:51:33.511579 11732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:51:33.511623 11732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:51:33.513171 11732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:41155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:51:33.513447 11732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:51:33.513698 11732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:51:33.518553 11740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:51:33.518690 11737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:51:33.518566 11738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:33.519009 11732 server_base.cc:1034] running on GCE node
I20260619 20:51:33.519238 11732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:51:33.519447 11732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:51:33.520605 11732 hybrid_clock.cc:648] HybridClock initialized: now 1781902293520590 us; error 34 us; skew 500 ppm
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:51:33.523204 11732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:51:33.523631 11732 webserver.cc:460] Webserver started at http://127.0.98.1:39265/ using document root <none> and password file <none>
I20260619 20:51:33.523852 11732 fs_manager.cc:359] Metadata directory not provided
I20260619 20:51:33.523919 11732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:51:33.524060 11732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:51:33.539083 11732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "9e26ef615fa94539829c72d576a9353e"
format_stamp: "Formatted at 2026-06-19 20:51:33 on dist-test-slave-0smx"
server_key: "239b2e16d65e33526294e7b14376a5f2"
server_key_iv: "31818b53813475c4b70000a3ecae8332"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:33.539662 11732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9e26ef615fa94539829c72d576a9353e"
format_stamp: "Formatted at 2026-06-19 20:51:33 on dist-test-slave-0smx"
server_key: "239b2e16d65e33526294e7b14376a5f2"
server_key_iv: "31818b53813475c4b70000a3ecae8332"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:33.545440 11732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.001s
I20260619 20:51:33.557755 11747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:33.557973 11732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:51:33.558075 11732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "9e26ef615fa94539829c72d576a9353e"
format_stamp: "Formatted at 2026-06-19 20:51:33 on dist-test-slave-0smx"
server_key: "239b2e16d65e33526294e7b14376a5f2"
server_key_iv: "31818b53813475c4b70000a3ecae8332"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:33.558175 11732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:51:33.580999 11732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:51:33.583379 11732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:51:33.583606 11732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:51:33.583921 11732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:51:33.584360 11732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:51:33.584447 11732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:33.584506 11732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:51:33.584547 11732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:33.591940 11732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35743
I20260619 20:51:33.592116 11860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35743 every 8 connection(s)
I20260619 20:51:33.592437 11732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:51:33.594162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 11732
I20260619 20:51:33.594267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:51:33.594460 392 external_mini_cluster.cc:1428] Setting key 09b1043cfc74197848becd9b695c8fd8
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:51:33.607613 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:33.593239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:33843)
0619 20:51:33.593483 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:33.593524 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:33.596930 (+ 3406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:33.597332 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:33.597337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:33.597603 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:51:33.597932 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:33.597939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.598991 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.598993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:33.600380 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:33.600384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.600970 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.600971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:33.601115 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:33.601747 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:51:33.601758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:51:33.603498 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:51:33.605981 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:51:33.605987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:51:33.605990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:51:33.606272 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:51:33.606506 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:51:33.606506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:51:33.606507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:51:33.606635 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:51:33.606916 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:51:33.606923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:51:33.607107 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 20:51:33.607201 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 20:51:33.607372 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
I20260619 20:51:33.608444 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:33.593249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33843 (local address 127.0.98.62:36285)
0619 20:51:33.593423 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:33.593426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:33.593612 (+ 186us) server_negotiation.cc:407] Connection header received
0619 20:51:33.597105 (+ 3493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:33.597106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:33.597121 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:33.597182 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:51:33.598055 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.598883 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.600496 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.600819 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.600895 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:33.603607 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:51:33.603609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:51:33.603609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:51:33.603617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:51:33.605889 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:33.606376 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:33.606377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:33.606378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:33.606441 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:33.606700 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:33.606701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:33.606701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:33.606830 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:51:33.606889 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:33.608247 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0619 20:51:33.608349 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260619 20:51:33.609193 11861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260619 20:51:33.609368 11861 heartbeater.cc:464] Registering TS with master...
I20260619 20:51:33.609592 11861 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260619 20:51:33.610208 11651 ts_manager.cc:194] Registered new tserver with Master: 9e26ef615fa94539829c72d576a9353e (127.0.98.1:35743)
I20260619 20:51:33.611085 11651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33843
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:51:33.634521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:51:33.749629 11869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:51:33.749928 11869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:51:33.749970 11869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:51:33.751755 11869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:51:33.751820 11869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:51:33.751864 11869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:51:33.754084 11869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:41155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:51:33.754361 11869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:51:33.754689 11869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:51:33.761783 11879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:33.762178 11869 server_base.cc:1034] running on GCE node
W20260619 20:51:33.762290 11876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:51:33.762356 11877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:33.762624 11869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:51:33.762923 11869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:51:33.764093 11869 hybrid_clock.cc:648] HybridClock initialized: now 1781902293764067 us; error 44 us; skew 500 ppm
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:51:33.766801 11869 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:51:33.767304 11869 webserver.cc:460] Webserver started at http://127.0.98.2:41731/ using document root <none> and password file <none>
I20260619 20:51:33.767565 11869 fs_manager.cc:359] Metadata directory not provided
I20260619 20:51:33.767635 11869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:51:33.767771 11869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:51:33.781273 11869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "06c1e11f7d3e41fcaa0c4796a168b114"
format_stamp: "Formatted at 2026-06-19 20:51:33 on dist-test-slave-0smx"
server_key: "6814d6621e82fd884e6248dac4bedccf"
server_key_iv: "3b3ca6bd8994aae3f718dacb4227bcd6"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:33.781773 11869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "06c1e11f7d3e41fcaa0c4796a168b114"
format_stamp: "Formatted at 2026-06-19 20:51:33 on dist-test-slave-0smx"
server_key: "6814d6621e82fd884e6248dac4bedccf"
server_key_iv: "3b3ca6bd8994aae3f718dacb4227bcd6"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:33.785684 11869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260619 20:51:33.808776 11886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:33.809089 11869 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 20:51:33.809229 11869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "06c1e11f7d3e41fcaa0c4796a168b114"
format_stamp: "Formatted at 2026-06-19 20:51:33 on dist-test-slave-0smx"
server_key: "6814d6621e82fd884e6248dac4bedccf"
server_key_iv: "3b3ca6bd8994aae3f718dacb4227bcd6"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:33.809376 11869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:51:33.852933 11869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:51:33.855648 11869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:51:33.855877 11869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:51:33.856138 11869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:51:33.856568 11869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:51:33.856665 11869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:33.856742 11869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:51:33.856805 11869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:33.869481 11869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37891
I20260619 20:51:33.870020 11869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:51:33.872272 11999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37891 every 8 connection(s)
I20260619 20:51:33.876027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 11869
I20260619 20:51:33.876137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:51:33.876389 392 external_mini_cluster.cc:1428] Setting key 423efc4834a8d7a2644862f0ee94f6e5
Jun 19 20:51:33 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:51:33.884625 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:33.873277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41147 (local address 127.0.98.62:36285)
0619 20:51:33.873373 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:33.873375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:33.873549 (+ 174us) server_negotiation.cc:407] Connection header received
0619 20:51:33.874288 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:33.874289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:33.874303 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:33.874340 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:51:33.875199 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.876051 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.877927 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.878268 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.878317 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:33.881029 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:51:33.881031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:51:33.881031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:51:33.881039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:51:33.882890 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:33.883450 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:33.883451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:33.883452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:33.883514 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:33.883813 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:33.883814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:33.883814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:33.883952 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:51:33.883998 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:33.884410 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 20:51:33.884501 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:51:33.885394 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:33.872975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:41147)
0619 20:51:33.873410 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:33.873457 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:33.874129 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:33.874540 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:33.874544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:33.874799 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:51:33.875069 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:33.875076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.876316 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.876318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:33.877814 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:33.877817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:33.878414 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:33.878415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:33.878569 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:33.879208 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:51:33.879219 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:51:33.880922 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:51:33.883005 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:51:33.883012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:51:33.883014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:51:33.883348 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:51:33.883596 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:51:33.883597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:51:33.883597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:51:33.883727 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:51:33.884049 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:51:33.884055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:51:33.884224 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 20:51:33.884965 (+ 741us) client_negotiation.cc:240] Negotiation successful
0619 20:51:33.885127 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":87,"threads_started":1}
I20260619 20:51:33.886125 12000 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260619 20:51:33.886238 12000 heartbeater.cc:464] Registering TS with master...
I20260619 20:51:33.886435 12000 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260619 20:51:33.886861 11651 ts_manager.cc:194] Registered new tserver with Master: 06c1e11f7d3e41fcaa0c4796a168b114 (127.0.98.2:37891)
I20260619 20:51:33.887367 11651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41147
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:51:33.916077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:51:34.039347 12006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:51:34.039714 12006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:51:34.039815 12006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:51:34.041608 12006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:51:34.041922 12006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:51:34.042025 12006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:51:34.044201 12006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:41155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:51:34.044708 12006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:51:34.045038 12006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:51:34.052018 12012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:51:34.052809 12011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:34.053260 12006 server_base.cc:1034] running on GCE node
W20260619 20:51:34.053440 12014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:51:34.057941 12006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:51:34.058230 12006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:51:34.065538 12006 hybrid_clock.cc:648] HybridClock initialized: now 1781902294065508 us; error 39 us; skew 500 ppm
Jun 19 20:51:34 dist-test-slave-0smx krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:51:34.075943 12006 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:51:34.076498 12006 webserver.cc:460] Webserver started at http://127.0.98.3:46731/ using document root <none> and password file <none>
I20260619 20:51:34.076795 12006 fs_manager.cc:359] Metadata directory not provided
I20260619 20:51:34.076859 12006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:51:34.076992 12006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:51:34 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:51:34.093369 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "debb85644bf440a799b3d2d3e1ec18fc"
format_stamp: "Formatted at 2026-06-19 20:51:34 on dist-test-slave-0smx"
server_key: "81e0c00977bb951db96b00a7e3ed31ca"
server_key_iv: "68333fde9ce4c2c3ac836a929642dcc0"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:34.094209 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "debb85644bf440a799b3d2d3e1ec18fc"
format_stamp: "Formatted at 2026-06-19 20:51:34 on dist-test-slave-0smx"
server_key: "81e0c00977bb951db96b00a7e3ed31ca"
server_key_iv: "68333fde9ce4c2c3ac836a929642dcc0"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:34.098394 12006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260619 20:51:34.127130 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:34.127440 12006 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 20:51:34.127552 12006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "debb85644bf440a799b3d2d3e1ec18fc"
format_stamp: "Formatted at 2026-06-19 20:51:34 on dist-test-slave-0smx"
server_key: "81e0c00977bb951db96b00a7e3ed31ca"
server_key_iv: "68333fde9ce4c2c3ac836a929642dcc0"
server_key_version: "kuduclusterkey@0"
I20260619 20:51:34.127671 12006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:51:34.148501 12006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:51:34.151628 12006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:51:34.151769 12006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:51:34.151969 12006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:51:34.152422 12006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:51:34.152475 12006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:34.152513 12006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:51:34.152536 12006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:51:34.161162 12006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40915
I20260619 20:51:34.161695 12006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:51:34.163599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 12006
I20260619 20:51:34.163695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:51:34.163905 392 external_mini_cluster.cc:1428] Setting key abcaea235d91bf3793412a8dc9c71be0
I20260619 20:51:34.170819 12134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40915 every 8 connection(s)
Jun 19 20:51:34 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:51:34.184422 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.171805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47515 (local address 127.0.98.62:36285)
0619 20:51:34.171854 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.171856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.173936 (+ 2080us) server_negotiation.cc:407] Connection header received
0619 20:51:34.174649 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.174651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.174665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.174699 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:51:34.175575 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.176372 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.177936 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.178208 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.178264 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.180977 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:51:34.180978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:51:34.180978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:51:34.180986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:51:34.182762 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:34.183268 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:34.183269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:34.183269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:34.183331 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:34.183614 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:34.183615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:34.183616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:34.183749 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:51:34.183795 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.184210 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.184317 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:51:34.185179 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.171514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:47515)
0619 20:51:34.173797 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.173848 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:34.174516 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:34.174896 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:34.174901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:34.175165 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:51:34.175457 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.175463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.176499 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.176500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.177828 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.177831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.178362 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.178362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.178501 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.179123 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:51:34.179133 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:51:34.180861 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:51:34.182866 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:51:34.182873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:51:34.182875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:51:34.183153 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:51:34.183408 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:51:34.183408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:51:34.183409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:51:34.183535 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:51:34.183862 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:51:34.183868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:51:34.184045 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 20:51:34.184803 (+ 758us) client_negotiation.cc:240] Negotiation successful
0619 20:51:34.184962 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2235,"thread_start_us":89,"threads_started":1}
I20260619 20:51:34.186048 12135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260619 20:51:34.186149 12135 heartbeater.cc:464] Registering TS with master...
I20260619 20:51:34.186348 12135 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260619 20:51:34.187059 11651 ts_manager.cc:194] Registered new tserver with Master: debb85644bf440a799b3d2d3e1ec18fc (127.0.98.3:40915)
I20260619 20:51:34.187579 11651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47515
I20260619 20:51:34.197778 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:51:34.208664 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.200002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.0.98.62:36285)
0619 20:51:34.200050 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.200052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.200350 (+ 298us) server_negotiation.cc:407] Connection header received
0619 20:51:34.200459 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.200461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.200472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.200506 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:51:34.201412 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.202225 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.203447 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.203733 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.203782 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.205556 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:51:34.205557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:51:34.205558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:51:34.205565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:51:34.207338 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:34.207750 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:34.207751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:34.207751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:34.207813 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:51:34.208027 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:51:34.208027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:51:34.208028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:51:34.208150 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:51:34.208243 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.208480 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.208562 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:51:34.210793 11651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:51:34.213124 11651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:51:34.227561 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.216856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35743 (local address 127.0.0.1:42432)
0619 20:51:34.218214 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.218222 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:34.218258 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:34.218873 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:34.218876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:34.218879 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:51:34.218949 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.218951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.222927 (+ 3976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.222928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.224088 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.224091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.226288 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.226289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.226378 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.226395 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 20:51:34.227372 (+ 977us) client_negotiation.cc:240] Negotiation successful
0619 20:51:34.227439 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"thread_start_us":59,"threads_started":1}
I20260619 20:51:34.228503 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.217584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.0.98.1:35743)
0619 20:51:34.218648 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.218653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.218662 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:51:34.218695 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.218697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.218712 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.218783 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:51:34.219138 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.222827 (+ 3689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.225340 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.226122 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.226197 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.228351 (+ 2154us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.228378 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.228425 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":48,"threads_started":1}
I20260619 20:51:34.229125 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.216650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40915 (local address 127.0.0.1:50450)
0619 20:51:34.218010 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.218053 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:34.218114 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:34.219024 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:34.219024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:34.219026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:51:34.219065 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.219066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.221022 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.221023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.225100 (+ 4077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.225103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.228873 (+ 3770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.228874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.228951 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.228955 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:51:34.229007 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:51:34.229047 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1329,"thread_start_us":63,"threads_started":1}
I20260619 20:51:34.229633 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.217410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.98.3:40915)
0619 20:51:34.218387 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.218393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.218403 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:51:34.218441 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.218443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.218459 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.218533 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:51:34.219847 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.220923 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.225217 (+ 4294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.227243 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.228758 (+ 1515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.228845 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.229505 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.229550 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":87,"threads_started":1}
I20260619 20:51:34.230474 11795 tablet_service.cc:1467] Processing CreateTablet for tablet 87678152932a4048833027deaf383f49 (DEFAULT_TABLE table=test-table [id=eab99d7aa0b344379032491a24e8a900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:51:34.230897 11795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87678152932a4048833027deaf383f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:51:34.235791 12064 tablet_service.cc:1467] Processing CreateTablet for tablet 87678152932a4048833027deaf383f49 (DEFAULT_TABLE table=test-table [id=eab99d7aa0b344379032491a24e8a900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:51:34.236150 12064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87678152932a4048833027deaf383f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:51:34.244351 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.217038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37891 (local address 127.0.0.1:41446)
0619 20:51:34.224213 (+ 7175us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.224224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:34.224266 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:34.240432 (+ 16166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:34.240433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:34.240437 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:51:34.240494 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.240496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.241913 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.241914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.243060 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.243063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.244084 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.244085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.244148 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.244155 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:51:34.244189 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:51:34.244255 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7158,"thread_start_us":74,"threads_started":1}
I20260619 20:51:34.244994 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.217168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.0.98.2:37891)
0619 20:51:34.240225 (+ 23057us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.240231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.240242 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:51:34.240297 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.240300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.240316 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.240572 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:51:34.240651 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.241803 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.243192 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.243975 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.244755 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.244845 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.244873 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.244910 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23012,"thread_start_us":101,"threads_started":1}
I20260619 20:51:34.245695 12154 tablet_bootstrap.cc:492] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: Bootstrap starting.
I20260619 20:51:34.245975 12155 tablet_bootstrap.cc:492] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Bootstrap starting.
I20260619 20:51:34.246980 11933 tablet_service.cc:1467] Processing CreateTablet for tablet 87678152932a4048833027deaf383f49 (DEFAULT_TABLE table=test-table [id=eab99d7aa0b344379032491a24e8a900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:51:34.247350 11933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87678152932a4048833027deaf383f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:51:34.252534 12155 tablet_bootstrap.cc:654] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Neither blocks nor log segments found. Creating new log.
I20260619 20:51:34.253578 12154 tablet_bootstrap.cc:654] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: Neither blocks nor log segments found. Creating new log.
I20260619 20:51:34.253957 12154 log.cc:824] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:51:34.254370 12155 log.cc:824] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:51:34.254937 12156 tablet_bootstrap.cc:492] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Bootstrap starting.
I20260619 20:51:34.255965 12156 tablet_bootstrap.cc:654] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Neither blocks nor log segments found. Creating new log.
I20260619 20:51:34.256345 12156 log.cc:824] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Log is configured to *not* fsync() on all Append() calls
I20260619 20:51:34.257136 12154 tablet_bootstrap.cc:492] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: No bootstrap required, opened a new log
I20260619 20:51:34.257251 12154 ts_tablet_manager.cc:1397] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260619 20:51:34.260051 12154 raft_consensus.cc:348] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.260336 12154 raft_consensus.cc:374] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:51:34.260433 12154 raft_consensus.cc:729] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e26ef615fa94539829c72d576a9353e, State: Initialized, Role: FOLLOWER
I20260619 20:51:34.260646 12154 consensus_queue.cc:260] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.261009 12154 ts_tablet_manager.cc:1428] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 20:51:34.261200 11861 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260619 20:51:34.261864 12155 tablet_bootstrap.cc:492] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: No bootstrap required, opened a new log
I20260619 20:51:34.261987 12155 ts_tablet_manager.cc:1397] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260619 20:51:34.264886 12155 raft_consensus.cc:348] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.265120 12155 raft_consensus.cc:374] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:51:34.265208 12155 raft_consensus.cc:729] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debb85644bf440a799b3d2d3e1ec18fc, State: Initialized, Role: FOLLOWER
I20260619 20:51:34.265439 12155 consensus_queue.cc:260] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.265774 12155 ts_tablet_manager.cc:1428] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 20:51:34.265934 12135 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260619 20:51:34.268091 12156 tablet_bootstrap.cc:492] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: No bootstrap required, opened a new log
I20260619 20:51:34.268213 12156 ts_tablet_manager.cc:1397] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260619 20:51:34.270377 12156 raft_consensus.cc:348] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.270612 12156 raft_consensus.cc:374] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:51:34.270700 12156 raft_consensus.cc:729] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1e11f7d3e41fcaa0c4796a168b114, State: Initialized, Role: FOLLOWER
I20260619 20:51:34.270910 12156 consensus_queue.cc:260] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.271220 12156 ts_tablet_manager.cc:1428] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:51:34.271389 12000 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
W20260619 20:51:34.343609 11862 tablet.cc:2369] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:51:34 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 20:51:34.373122 12001 tablet.cc:2369] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:51:34.415707 12136 tablet.cc:2369] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:51:34.560303 12162 raft_consensus.cc:482] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:51:34.560453 12162 raft_consensus.cc:504] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.560843 12162 leader_election.cc:288] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers debb85644bf440a799b3d2d3e1ec18fc (127.0.98.3:40915), 9e26ef615fa94539829c72d576a9353e (127.0.98.1:35743)
I20260619 20:51:34.564662 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.561115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34721 (local address 127.0.98.3:40915)
0619 20:51:34.561163 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.561165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.561321 (+ 156us) server_negotiation.cc:407] Connection header received
0619 20:51:34.561423 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.561425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.561435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.561462 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:51:34.561667 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.562408 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.563749 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.564292 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.564337 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.564346 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.564557 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.564594 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:51:34.565058 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.561045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40915 (local address 127.0.98.2:34721)
0619 20:51:34.561229 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.561241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:34.561350 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:34.561528 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:34.561528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:34.561530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:51:34.561593 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.561595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.562523 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.562524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.563650 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.563653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.564403 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.564404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.564471 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.564484 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:51:34.564941 (+ 457us) client_negotiation.cc:240] Negotiation successful
0619 20:51:34.564986 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260619 20:51:34.565500 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87678152932a4048833027deaf383f49" candidate_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debb85644bf440a799b3d2d3e1ec18fc" is_pre_election: true
I20260619 20:51:34.565644 12089 raft_consensus.cc:2454] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1e11f7d3e41fcaa0c4796a168b114 in term 0.
I20260619 20:51:34.565953 11888 leader_election.cc:302] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1e11f7d3e41fcaa0c4796a168b114, debb85644bf440a799b3d2d3e1ec18fc; no voters:
I20260619 20:51:34.566072 12162 raft_consensus.cc:2781] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:51:34.566105 12162 raft_consensus.cc:482] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:51:34.566125 12162 raft_consensus.cc:3037] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:51:34.566947 12162 raft_consensus.cc:504] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.567063 12162 leader_election.cc:288] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [CANDIDATE]: Term 1 election: Requested vote from peers debb85644bf440a799b3d2d3e1ec18fc (127.0.98.3:40915), 9e26ef615fa94539829c72d576a9353e (127.0.98.1:35743)
I20260619 20:51:34.567389 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87678152932a4048833027deaf383f49" candidate_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debb85644bf440a799b3d2d3e1ec18fc"
I20260619 20:51:34.567459 12089 raft_consensus.cc:3037] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:51:34.568316 12089 raft_consensus.cc:2454] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1e11f7d3e41fcaa0c4796a168b114 in term 1.
I20260619 20:51:34.568635 11888 leader_election.cc:302] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1e11f7d3e41fcaa0c4796a168b114, debb85644bf440a799b3d2d3e1ec18fc; no voters:
I20260619 20:51:34.568755 12162 raft_consensus.cc:2781] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:51:34.568877 12162 raft_consensus.cc:686] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 1 LEADER]: Becoming Leader. State: Replica: 06c1e11f7d3e41fcaa0c4796a168b114, State: Running, Role: LEADER
I20260619 20:51:34.568964 12162 consensus_queue.cc:237] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } }
I20260619 20:51:34.569813 11651 catalog_manager.cc:5495] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1e11f7d3e41fcaa0c4796a168b114 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 } health_report { overall_health: UNKNOWN } } }
I20260619 20:51:34.570866 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.561395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46005 (local address 127.0.98.1:35743)
0619 20:51:34.561437 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.561439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.561703 (+ 264us) server_negotiation.cc:407] Connection header received
0619 20:51:34.561856 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.561857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.561870 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.561900 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:51:34.562124 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.562834 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.569870 (+ 7036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.570457 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.570511 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.570521 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.570749 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.570800 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:51:34.571254 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.561238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35743 (local address 127.0.98.2:46005)
0619 20:51:34.561633 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.561644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:51:34.561772 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:51:34.561968 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:51:34.561968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:51:34.561970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:51:34.562040 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.562043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.568569 (+ 6526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.568569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.569755 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:51:34.569758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.570583 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.570583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:51:34.570657 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.570662 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:51:34.571100 (+ 438us) client_negotiation.cc:240] Negotiation successful
0619 20:51:34.571149 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":228,"threads_started":1}
I20260619 20:51:34.571691 11815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87678152932a4048833027deaf383f49" candidate_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e26ef615fa94539829c72d576a9353e" is_pre_election: true
I20260619 20:51:34.571703 11814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87678152932a4048833027deaf383f49" candidate_uuid: "06c1e11f7d3e41fcaa0c4796a168b114" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e26ef615fa94539829c72d576a9353e"
I20260619 20:51:34.571861 11815 raft_consensus.cc:2454] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1e11f7d3e41fcaa0c4796a168b114 in term 0.
Jun 19 20:51:34 dist-test-slave-0smx krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:51:34.663365 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.659994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.98.2:37891)
0619 20:51:34.660046 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:51:34.660049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:51:34.660059 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:51:34.660107 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:51:34.660108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:51:34.660132 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:51:34.660221 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:51:34.660377 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.661083 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.662192 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:51:34.662520 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:51:34.662587 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:51:34.662785 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:51:34.662918 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:51:34.663006 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:51:34.663235 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 20:51:34.663253 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:51:34.665493 11814 raft_consensus.cc:3037] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:51:34.666508 11814 raft_consensus.cc:1261] T 87678152932a4048833027deaf383f49 P 9e26ef615fa94539829c72d576a9353e [term 1 FOLLOWER]: Refusing update from remote peer 06c1e11f7d3e41fcaa0c4796a168b114: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:51:34.667198 12089 raft_consensus.cc:1261] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 1 FOLLOWER]: Refusing update from remote peer 06c1e11f7d3e41fcaa0c4796a168b114: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:51:34.667407 12165 consensus_queue.cc:1035] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e26ef615fa94539829c72d576a9353e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:51:34.667860 12165 consensus_queue.cc:1035] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debb85644bf440a799b3d2d3e1ec18fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:51:34.675125 12170 mvcc.cc:205] Tried to move back new op lower bound from 7298671798947508224 to 7298671798555009024. Current Snapshot: MvccSnapshot[applied={T|T < 7298671798947508224}]
I20260619 20:51:34.679121 11651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:51:34.679215 11651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:51:34.680080 11651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5bc8f40f2644cb1895443fbc634bc30: Sending DeleteTablet for 3 replicas of tablet 87678152932a4048833027deaf383f49
I20260619 20:51:34.680588 12064 tablet_service.cc:1514] Processing DeleteTablet for tablet 87678152932a4048833027deaf383f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:51:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260619 20:51:34.680827 11795 tablet_service.cc:1514] Processing DeleteTablet for tablet 87678152932a4048833027deaf383f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:51:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42432
I20260619 20:51:34.681428 11933 tablet_service.cc:1514] Processing DeleteTablet for tablet 87678152932a4048833027deaf383f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:51:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41446
I20260619 20:51:34.682582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 11732
I20260619 20:51:34.684239 12171 mvcc.cc:205] Tried to move back new op lower bound from 7298671798947508224 to 7298671798555009024. Current Snapshot: MvccSnapshot[applied={T|T < 7298671798947508224}]
I20260619 20:51:34.685559 12177 tablet_replica.cc:330] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: stopping tablet replica
I20260619 20:51:34.685714 12177 raft_consensus.cc:2229] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:51:34.691685 12177 raft_consensus.cc:2258] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:51:34.692314 12177 ts_tablet_manager.cc:1905] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:51:34.694062 12177 ts_tablet_manager.cc:1918] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:51:34.694195 12177 log.cc:1196] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/87678152932a4048833027deaf383f49
I20260619 20:51:34.694535 12177 ts_tablet_manager.cc:1939] T 87678152932a4048833027deaf383f49 P debb85644bf440a799b3d2d3e1ec18fc: Deleting consensus metadata
I20260619 20:51:34.695089 11636 catalog_manager.cc:4841] TS debb85644bf440a799b3d2d3e1ec18fc (127.0.98.3:40915): tablet 87678152932a4048833027deaf383f49 (table test-table [id=eab99d7aa0b344379032491a24e8a900]) successfully deleted
I20260619 20:51:34.704212 12179 tablet_replica.cc:330] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: stopping tablet replica
I20260619 20:51:34.704376 12179 raft_consensus.cc:2229] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:51:34.704495 12179 raft_consensus.cc:2258] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:51:34.705024 12179 ts_tablet_manager.cc:1905] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:51:34.705487 11636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35743 (error 108)
I20260619 20:51:34.706828 12179 ts_tablet_manager.cc:1918] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:51:34.706940 12179 log.cc:1196] T 87678152932a4048833027deaf383f49 P 06c1e11f7d3e41fcaa0c4796a168b114: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/87678152932a4048833027deaf383f49
I20260619 20:51:34.707053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 11869
I20260619 20:51:34.720475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 12006
I20260619 20:51:34.720582 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.720448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35743 (local address 127.0.0.1:48014)
0619 20:51:34.720504 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.720545 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:51:34.722457 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:51:34.722359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37891 (local address 127.0.0.1:32862)
0619 20:51:34.722407 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:51:34.722429 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:51:34.722637 11636 catalog_manager.cc:4584] TS 9e26ef615fa94539829c72d576a9353e (127.0.98.1:35743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87678152932a4048833027deaf383f49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35743: connect: Connection refused (error 111)
I20260619 20:51:34.726748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 11619
I20260619 20:51:34.737751 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 20:51:35.082260 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 20:51:35.082381 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 20:51:35.426643 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 20:51:35.426 UTC [11205] LOG: received smart shutdown request
2026-06-19 20:51:35.429 UTC [11205] LOG: background worker "logical replication launcher" (PID 11214) exited with exit code 1
2026-06-19 20:51:35.430 UTC [11209] LOG: shutting down
2026-06-19 20:51:35.582 UTC [11205] LOG: database system is shut down
2026-06-19T20:51:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45476 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:51:35 dist-test-slave-0smx krb5kdc[12190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:51:35 dist-test-slave-0smx krb5kdc[12190](info): set up 2 sockets
Jun 19 20:51:35 dist-test-slave-0smx krb5kdc[12190](info): commencing operation
krb5kdc: starting...
W20260619 20:51:36.725068 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:51:36 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:51:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 20:51:36.875473 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskH2GnNK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/postgres -l logfile start
2026-06-19 20:51:39.200 UTC [12220] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 20:51:39.200 UTC [12220] LOG: listening on IPv4 address "127.0.98.20", port 58025
2026-06-19 20:51:39.203 UTC [12220] LOG: listening on Unix socket "/tmp/.s.PGSQL.58025"
2026-06-19 20:51:39.223 UTC [12223] LOG: database system was shut down at 2026-06-19 20:51:37 UTC
2026-06-19 20:51:39.227 UTC [12220] LOG: database system is ready to accept connections
I20260619 20:51:40.173175 392 mini_postgres.cc:96] Postgres bound to 58025
2026-06-19 20:51:40.178 UTC [12231] FATAL: database "slave" does not exist
127.0.98.20:58025 - accepting connections
I20260619 20:51:40.179477 392 mini_ranger.cc:161] Starting Ranger...
I20260619 20:51:40.193423 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 20:51:40.637427 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 20:51:40.637511 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-admin
2026-06-19 20:51:40,908 [I] DB FLAVOR :POSTGRES
2026-06-19 20:51:40,908 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:51:40,908 [I] Checking connection..
2026-06-19 20:51:40,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 20:51:41,147 [I] Checking connection passed.
2026-06-19 20:51:41,148 [I] --------- Verifying version history table ---------
2026-06-19 20:51:41,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:51:41,373 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:51:41,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:51:41,629 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:51:41,630 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 20:51:41,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 20:51:41.869 UTC [12295] WARNING: there is no transaction in progress
2026-06-19 20:51:41,875 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 20:51:41,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:51:42,128 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 20:51:42,128 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:51:42,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 20:51:42,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 20:51:42,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0smx.c.gcp-upstream.internal','N') ;"
2026-06-19 20:51:42,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:51:43,112 [I] Table x_portal_user does not exist in database ranger
2026-06-19 20:51:43,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:51:43,355 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 20:51:43,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:51:43,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:51:43,813 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 20:51:43,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 20:51:44.747 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.764 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.779 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.794 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.809 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.826 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.881 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.888 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.902 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.913 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.923 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.937 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.946 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.952 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.959 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:44.965 UTC [12403] WARNING: there is no transaction in progress
2026-06-19 20:51:45,575 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 20:51:45,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 20:51:45,800 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 20:51:45,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:51:46,047 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 20:51:46,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:51:46,305 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 20:51:46,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:51:46,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:51:46,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0smx.c.gcp-upstream.internal';"
2026-06-19 20:51:47,031 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 20:51:47,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 20:51:47,274 [I] DB_PATCHES have already been applied
I20260619 20:51:47.280297 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskH2GnNK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 20:51:47.280386 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 20:51:47.283234 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44275
Jun 19, 2026 8:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:51:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:51:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 20:51:48.382191 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.203s user 0.001s sys 0.007s
Jun 19, 2026 8:52:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:52:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 8:52:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:52:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:52:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 20:52:08.935535 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 20:52:09 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902329, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 20:52:09.013512 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 20:52:09.022337 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 20:52:09.843576 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 20:52:09.843644 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms
I20260619 20:52:09.843657 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58025
2026-06-19 20:52:10,003 [I] DB FLAVOR :POSTGRES
2026-06-19 20:52:10,004 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:52:10,004 [I] Checking connection
2026-06-19 20:52:10,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 20:52:10,257 [I] connection success
2026-06-19 20:52:10,258 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 20:52:10,258 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 20:52:10,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 20:52:10,484 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 20:52:10,485 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:52:10,485 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 20:52:10,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 20:52:10,749 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 20:52:11.201335 392 mini_ranger_kms.cc:310] Created kms service
I20260619 20:52:11.402164 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 20:52:11.449416 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 20:52:11 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902329, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:11.893887 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 20:52:11.894001 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskH2GnNK/build/release/bin/postgresql.jar:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskH2GnNK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 20:52:11.894025 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 20:52:11.897356 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12583
I20260619 20:52:11.897434 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44129
Jun 19, 2026 8:52:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:52:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:52:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:52:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:52:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:52:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 20:52:14.146577 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.133s user 0.000s sys 0.142s
I20260619 20:52:14.146693 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 20:52:14.146745 392 mini_ranger_kms.cc:401] 127.0.98.20:44129/kms/v1/keys
Jun 19 20:52:14 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902334, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 20:52:14 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902334, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:52:14 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902334, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 20:52:17 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:52:17.308755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45681
--ranger_config_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:17.414529 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:17.414729 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:17.414752 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:17.415879 12634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:17.415920 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:17.415933 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:17.415946 12634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:17.415998 12634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:17.417580 12634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44129/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45681
--ranger_config_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:17.417755 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:17.417963 12634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:17.422868 12640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:17.422840 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:17.422839 12639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:17.422855 12634 server_base.cc:1034] running on GCE node
I20260619 20:52:17.423374 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:17.423678 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:17.424827 12634 hybrid_clock.cc:648] HybridClock initialized: now 1781902337424807 us; error 34 us; skew 500 ppm
Jun 19 20:52:17 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:17.430163 12634 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:17.430598 12634 webserver.cc:460] Webserver started at http://127.0.98.62:36729/ using document root <none> and password file <none>
I20260619 20:52:17.430830 12634 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:17.430891 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:17.431046 12634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:52:17 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:17.475945 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee"
format_stamp: "Formatted at 2026-06-19 20:52:17 on dist-test-slave-0smx"
server_key: "3d662fdbb95f71775cfb78776e176756"
server_key_iv: "3c23991ac6afedd64a41798636bd0cd5"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:17.476383 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee"
format_stamp: "Formatted at 2026-06-19 20:52:17 on dist-test-slave-0smx"
server_key: "3d662fdbb95f71775cfb78776e176756"
server_key_iv: "3c23991ac6afedd64a41798636bd0cd5"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:17.478348 12634 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 20:52:17.493422 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:17.493654 12634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:52:17.493753 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee"
format_stamp: "Formatted at 2026-06-19 20:52:17 on dist-test-slave-0smx"
server_key: "3d662fdbb95f71775cfb78776e176756"
server_key_iv: "3c23991ac6afedd64a41798636bd0cd5"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:17.493854 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:17.505025 12634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:17.510988 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:17.511137 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:17.516029 12634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45681
I20260619 20:52:17.516058 12701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45681 every 8 connection(s)
I20260619 20:52:17.516461 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:17.517170 12702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:17.517402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 12634
I20260619 20:52:17.517488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:52:17.517665 392 external_mini_cluster.cc:1428] Setting key 174c05f193755b5d76d1525d443d4d7c
I20260619 20:52:17.520663 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Bootstrap starting.
I20260619 20:52:17.521323 12702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:17.521616 12702 log.cc:824] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:17.522315 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: No bootstrap required, opened a new log
I20260619 20:52:17.524447 12702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:17.524597 12702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:17.524670 12702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac1ed225f8844a5be4d7a0d7b6f7eee, State: Initialized, Role: FOLLOWER
I20260619 20:52:17.524797 12702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:17.524875 12702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:17.524904 12702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:17.524981 12702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 0 FOLLOWER]: Advancing to term 1
Jun 19 20:52:17 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:17.525699 12702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:17.525826 12702 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ac1ed225f8844a5be4d7a0d7b6f7eee; no voters:
I20260619 20:52:17.525983 12702 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:52:17.526326 12702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:17.526010 12707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:17.527812 12707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 LEADER]: Becoming Leader. State: Replica: 3ac1ed225f8844a5be4d7a0d7b6f7eee, State: Running, Role: LEADER
I20260619 20:52:17.527968 12707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:17.528764 12702 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:52:17.529081 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } } }
I20260619 20:52:17.529155 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:17.529335 12719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac1ed225f8844a5be4d7a0d7b6f7eee. Latest consensus state: current_term: 1 leader_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } } }
I20260619 20:52:17.529405 12719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:17.531618 12723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:17.532653 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:17.518194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.0.98.62:45681)
0619 20:52:17.518358 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:17.518367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:17.518398 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:52:17.519044 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:17.519054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:17.519115 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:17.519278 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:17.520022 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:17.521412 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:17.522505 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:17.523113 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:17.523250 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:17.526113 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:17.526118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:17.526118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:17.526141 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:17.528280 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:17.528684 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:17.528687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:17.528688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:17.528783 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:17.528993 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:17.528994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:17.528994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:17.529347 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:17.529500 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:17.531418 (+ 1918us) server_negotiation.cc:299] Negotiation successful
0619 20:52:17.532322 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260619 20:52:17.532750 12723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:17.534845 12723 catalog_manager.cc:1347] Generated new cluster ID: 28d72d4551ab4aee8e1b9f27b05e09c1
I20260619 20:52:17.534896 12723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:52:17.574271 12723 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:52:17.575114 12723 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:52:17.585312 12723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Generated new TSK 0
I20260619 20:52:17.585541 12723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:52:18.262239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45681
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:18.366052 12747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:18.366273 12747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:18.366312 12747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:18.367444 12747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:18.367514 12747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:18.367558 12747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:52:18.369158 12747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:18.369463 12747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:18.369689 12747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:18.374810 12753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:18.374825 12755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:18.375142 12752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:18.375277 12747 server_base.cc:1034] running on GCE node
I20260619 20:52:18.375488 12747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:18.375772 12747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:18.376967 12747 hybrid_clock.cc:648] HybridClock initialized: now 1781902338376954 us; error 26 us; skew 500 ppm
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:18.379667 12747 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:52:18.380157 12747 webserver.cc:460] Webserver started at http://127.0.98.1:40317/ using document root <none> and password file <none>
I20260619 20:52:18.380417 12747 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:18.380470 12747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:18.380607 12747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:18.394155 12747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "3b8ea01f0b5d4eaca171508795c1a1a8"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "318cb0c9aa3dd6c09670b43944030cc2"
server_key_iv: "9af34c0cfc98380822f82987748b079c"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.394549 12747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3b8ea01f0b5d4eaca171508795c1a1a8"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "318cb0c9aa3dd6c09670b43944030cc2"
server_key_iv: "9af34c0cfc98380822f82987748b079c"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.396212 12747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:52:18.408521 12763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.408806 12747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:52:18.408941 12747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "3b8ea01f0b5d4eaca171508795c1a1a8"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "318cb0c9aa3dd6c09670b43944030cc2"
server_key_iv: "9af34c0cfc98380822f82987748b079c"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.409032 12747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:18.431813 12747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:18.434492 12747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:18.434661 12747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:18.434919 12747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:18.435312 12747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:18.435364 12747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.435398 12747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:18.435457 12747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.442978 12747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43283
I20260619 20:52:18.443439 12747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:52:18.443776 12877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43283 every 8 connection(s)
I20260619 20:52:18.448051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 12747
I20260619 20:52:18.448156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:52:18.448359 392 external_mini_cluster.cc:1428] Setting key 1ba69ae38017fceabc5a9e136e2926e8
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:18.457717 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:18.444170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45681 (local address 127.0.98.1:57807)
0619 20:52:18.444319 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:18.444365 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:18.444978 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:18.445618 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:18.445622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:18.445864 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:18.446150 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:18.446156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.447171 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.447172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:18.450930 (+ 3758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:18.450933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.451489 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.451490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:18.451646 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:18.452263 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:18.452274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:18.453970 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:18.456064 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:18.456071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:18.456074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:18.456376 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:18.456605 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:18.456605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:18.456606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:18.456741 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:18.457020 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:18.457025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:18.457211 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 20:52:18.457323 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:52:18.457481 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260619 20:52:18.458490 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:18.444466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57807 (local address 127.0.98.62:45681)
0619 20:52:18.445379 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:18.445382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:18.445391 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:52:18.445418 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:18.445419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:18.445431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:18.446244 (+ 813us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:18.446291 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.447043 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.451040 (+ 3997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.451362 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.454065 (+ 2703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:18.454117 (+ 52us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:18.454118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:18.454118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:18.454123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:18.455964 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:18.456465 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:18.456466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:18.456467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:18.456534 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:18.456814 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:18.456815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:18.456815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:18.456945 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:18.458261 (+ 1316us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:18.458304 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 20:52:18.458389 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":77,"threads_started":1}
I20260619 20:52:18.459034 12878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45681
I20260619 20:52:18.459123 12878 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:18.459312 12878 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:52:18.459867 12666 ts_manager.cc:194] Registered new tserver with Master: 3b8ea01f0b5d4eaca171508795c1a1a8 (127.0.98.1:43283)
I20260619 20:52:18.460827 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:52:18.480569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45681
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:18.585775 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:18.586011 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:18.586068 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:18.587385 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:18.587443 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:18.587482 12884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:52:18.588912 12884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:18.589155 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:18.589460 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:18.594300 12894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:18.594338 12892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:18.594441 12891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:18.594508 12884 server_base.cc:1034] running on GCE node
I20260619 20:52:18.594902 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:18.595185 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:18.596354 12884 hybrid_clock.cc:648] HybridClock initialized: now 1781902338596325 us; error 38 us; skew 500 ppm
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:18.599208 12884 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:52:18.599694 12884 webserver.cc:460] Webserver started at http://127.0.98.2:38497/ using document root <none> and password file <none>
I20260619 20:52:18.599931 12884 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:18.600003 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:18.600137 12884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:18.614890 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "5a0d12e932234cdb99f126d269a8e496"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "75b5356a1f73483056fa52e4283cde48"
server_key_iv: "f1f787e3a7acd6554f6b0913a225b4a2"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.615439 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5a0d12e932234cdb99f126d269a8e496"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "75b5356a1f73483056fa52e4283cde48"
server_key_iv: "f1f787e3a7acd6554f6b0913a225b4a2"
server_key_version: "kuduclusterkey@0"
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:18.625779 12884 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260619 20:52:18.645476 12901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.645735 12884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:18.645828 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "5a0d12e932234cdb99f126d269a8e496"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "75b5356a1f73483056fa52e4283cde48"
server_key_iv: "f1f787e3a7acd6554f6b0913a225b4a2"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.645920 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:18.661851 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:18.664498 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:18.664717 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:18.665006 12884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:18.665452 12884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:18.665560 12884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.665637 12884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:18.665784 12884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.672293 12884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38643
I20260619 20:52:18.672792 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:52:18.675793 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38643 every 8 connection(s)
I20260619 20:52:18.679831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 12884
I20260619 20:52:18.679915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:52:18.680099 392 external_mini_cluster.cc:1428] Setting key 5f9f1f403559621a7cd078ce0216f462
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:18.688066 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:18.676797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41539 (local address 127.0.98.62:45681)
0619 20:52:18.676864 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:18.676866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:18.677156 (+ 290us) server_negotiation.cc:407] Connection header received
0619 20:52:18.677901 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:18.677902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:18.677914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:18.677945 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:18.678798 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.679588 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.681780 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.682069 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.682114 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:18.684753 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:18.684754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:18.684755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:18.684762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:18.686438 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:18.686941 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:18.686942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:18.686943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:18.687003 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:18.687289 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:18.687290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:18.687291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:18.687430 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:18.687486 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:18.687878 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 20:52:18.687968 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:52:18.688799 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:18.676507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45681 (local address 127.0.98.2:41539)
0619 20:52:18.677021 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:18.677068 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:18.677771 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:18.678135 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:18.678139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:18.678387 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:18.678680 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:18.678687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.680393 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.680394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:18.681679 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:18.681682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.682217 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.682218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:18.682350 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:18.682969 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:18.682980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:18.684642 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:18.686544 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:18.686551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:18.686553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:18.686847 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:18.687073 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:18.687074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:18.687074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:18.687210 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:18.687538 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:18.687544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:18.687719 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 20:52:18.688396 (+ 677us) client_negotiation.cc:240] Negotiation successful
0619 20:52:18.688568 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":82,"threads_started":1}
I20260619 20:52:18.689549 13015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45681
I20260619 20:52:18.689642 13015 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:18.689819 13015 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:18.690234 12666 ts_manager.cc:194] Registered new tserver with Master: 5a0d12e932234cdb99f126d269a8e496 (127.0.98.2:38643)
I20260619 20:52:18.690724 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41539
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:52:18.712870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45681
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:18.822019 13020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:18.822363 13020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:18.822464 13020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:18.824187 13020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:18.824292 13020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:18.824404 13020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:52:18.826612 13020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:18.827067 13020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:18.827383 13020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:18.836139 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:18.836486 13027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:18.836709 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:18.843890 13020 server_base.cc:1034] running on GCE node
I20260619 20:52:18.844236 13020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:18.844534 13020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:18.845741 13020 hybrid_clock.cc:648] HybridClock initialized: now 1781902338845719 us; error 37 us; skew 500 ppm
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:18.849048 13020 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:52:18.849676 13020 webserver.cc:460] Webserver started at http://127.0.98.3:33521/ using document root <none> and password file <none>
I20260619 20:52:18.849969 13020 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:18.850075 13020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:18.850245 13020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:18.872954 13020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf983e24544b4d838bc31b3a05279555"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "9a93904362175b03722100a7d25b290f"
server_key_iv: "99d2b62fa14de8542e8db345349c32f7"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.873613 13020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf983e24544b4d838bc31b3a05279555"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "9a93904362175b03722100a7d25b290f"
server_key_iv: "99d2b62fa14de8542e8db345349c32f7"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.876232 13020 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:18.898108 13036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.898370 13020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:18.898471 13020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "bf983e24544b4d838bc31b3a05279555"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "9a93904362175b03722100a7d25b290f"
server_key_iv: "99d2b62fa14de8542e8db345349c32f7"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:18.898568 13020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:18.931986 13020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:18.934639 13020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:18.934841 13020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:18.935134 13020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:18.935554 13020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:18.935640 13020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.935710 13020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:18.935765 13020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:18.942085 13020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39725
I20260619 20:52:18.942670 13020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:52:18.943302 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39725 every 8 connection(s)
I20260619 20:52:18.944231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13020
I20260619 20:52:18.944327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:52:18.944536 392 external_mini_cluster.cc:1428] Setting key b0b9ba69483d7129580b2a8df8710325
Jun 19 20:52:18 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:18.952946 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:18.944029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43493 (local address 127.0.98.62:45681)
0619 20:52:18.944092 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:18.944095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:18.944533 (+ 438us) server_negotiation.cc:407] Connection header received
0619 20:52:18.945374 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:18.945375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:18.945386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:18.945416 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:18.946055 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.946551 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.947565 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.947774 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.947872 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:18.950042 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:18.950044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:18.950044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:18.950051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:18.951572 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:18.951980 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:18.951982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:18.951983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:18.952052 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:18.952305 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:18.952306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:18.952307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:18.952416 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:18.952498 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:18.952785 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 20:52:18.952858 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:52:18.953176 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:18.944046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45681 (local address 127.0.98.3:43493)
0619 20:52:18.944307 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:18.944353 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:18.945231 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:18.945585 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:18.945588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:18.945771 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:18.945962 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:18.945965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.946644 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.946645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:18.947457 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:18.947459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:18.947891 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:18.947892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:18.948008 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:18.948549 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:18.948557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:18.949946 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:18.951655 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:18.951662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:18.951664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:18.951885 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:18.952125 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:18.952126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:18.952126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:18.952236 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:18.952489 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:18.952495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:18.952648 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 20:52:18.952753 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:52:18.952898 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260619 20:52:18.953764 13150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45681
I20260619 20:52:18.953884 13150 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:18.954068 13150 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:18.954468 12666 ts_manager.cc:194] Registered new tserver with Master: bf983e24544b4d838bc31b3a05279555 (127.0.98.3:39725)
I20260619 20:52:18.954856 12666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43493
I20260619 20:52:18.956033 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:52:18.956635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 12747
I20260619 20:52:18.962538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 12884
I20260619 20:52:18.973485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13020
I20260619 20:52:18.988938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 12634
I20260619 20:52:18.996215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45681
--webserver_interface=127.0.98.62
--webserver_port=36729
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45681
--ranger_config_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:19.094753 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:19.094959 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:19.094980 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:19.096086 13153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:19.096127 13153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:19.096141 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:19.096155 13153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:19.096167 13153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:19.097792 13153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45681
--ranger_config_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:19.098022 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:19.098258 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:19.103953 13158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:19.103978 13159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:19.104146 13161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:19.104220 13153 server_base.cc:1034] running on GCE node
I20260619 20:52:19.104466 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:19.104758 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:19.105894 13153 hybrid_clock.cc:648] HybridClock initialized: now 1781902339105881 us; error 31 us; skew 500 ppm
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:19.108362 13153 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:19.108850 13153 webserver.cc:460] Webserver started at http://127.0.98.62:36729/ using document root <none> and password file <none>
I20260619 20:52:19.109086 13153 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:19.109138 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:19.151538 13153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:19.152305 13169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.152503 13153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:19.152619 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee"
format_stamp: "Formatted at 2026-06-19 20:52:17 on dist-test-slave-0smx"
server_key: "3d662fdbb95f71775cfb78776e176756"
server_key_iv: "3c23991ac6afedd64a41798636bd0cd5"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:19.153082 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:19.174233 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:19.176151 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:19.176275 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:19.180478 13153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45681
I20260619 20:52:19.180521 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45681 every 8 connection(s)
I20260619 20:52:19.180929 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:19.182220 13222 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:52:19.182492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13153
I20260619 20:52:19.182801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43283
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45681
--webserver_port=40317
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:19.183219 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Bootstrap starting.
I20260619 20:52:19.193817 13222 log.cc:824] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:19.196050 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:52:19.196349 13222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Bootstrap complete.
I20260619 20:52:19.199707 13222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:19.200033 13222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac1ed225f8844a5be4d7a0d7b6f7eee, State: Initialized, Role: FOLLOWER
I20260619 20:52:19.200199 13222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:19.200270 13222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:19.200309 13222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:19.200357 13222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:52:19.201721 13222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:19.201941 13222 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ac1ed225f8844a5be4d7a0d7b6f7eee; no voters:
I20260619 20:52:19.202119 13222 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:52:19.202214 13226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:52:19.202354 13222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:19.202451 13226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [term 2 LEADER]: Becoming Leader. State: Replica: 3ac1ed225f8844a5be4d7a0d7b6f7eee, State: Running, Role: LEADER
I20260619 20:52:19.203703 13226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } }
I20260619 20:52:19.204236 13228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac1ed225f8844a5be4d7a0d7b6f7eee. Latest consensus state: current_term: 2 leader_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } } }
I20260619 20:52:19.204332 13228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:19.204401 13222 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:52:19.204545 13241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:19.204629 13227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac1ed225f8844a5be4d7a0d7b6f7eee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45681 } } }
I20260619 20:52:19.204679 13227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:19.205103 13241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:19.205228 13241 catalog_manager.cc:1259] Loaded cluster ID: 28d72d4551ab4aee8e1b9f27b05e09c1
I20260619 20:52:19.205265 13241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:52:19.206410 13241 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:52:19.206581 13241 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Loaded TSK: 0
I20260619 20:52:19.208874 13241 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:52:19.296077 13224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:19.296375 13224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:19.296427 13224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:19.297732 13224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:19.297787 13224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:19.297832 13224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:52:19.299721 13224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40317
--tserver_master_addrs=127.0.98.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:19.299974 13224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:19.300199 13224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:19.306296 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:19.306638 13224 server_base.cc:1034] running on GCE node
W20260619 20:52:19.306308 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:19.307199 13262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:19.307444 13224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:19.307713 13224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:19.308871 13224 hybrid_clock.cc:648] HybridClock initialized: now 1781902339308858 us; error 29 us; skew 500 ppm
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:19.311481 13224 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:52:19.311966 13224 webserver.cc:460] Webserver started at http://127.0.98.1:40317/ using document root <none> and password file <none>
I20260619 20:52:19.312234 13224 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:19.312310 13224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:19.339154 13224 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260619 20:52:19.340159 13270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.340553 13224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:19.340653 13224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "3b8ea01f0b5d4eaca171508795c1a1a8"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "318cb0c9aa3dd6c09670b43944030cc2"
server_key_iv: "9af34c0cfc98380822f82987748b079c"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:19.341146 13224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:19.366308 13224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:19.368947 13224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:19.369086 13224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:19.369345 13224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:19.369782 13224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:19.369827 13224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.369875 13224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:19.369900 13224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.378044 13224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43283
I20260619 20:52:19.378146 13383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43283 every 8 connection(s)
I20260619 20:52:19.378496 13224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:52:19.381532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13224
I20260619 20:52:19.381860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38643
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45681
--webserver_port=38497
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:19.393146 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.379308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45681 (local address 127.0.98.1:52071)
0619 20:52:19.379492 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:19.379537 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:19.380103 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:19.380866 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:19.380870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:19.381132 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:19.381446 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:19.381453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.383202 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.383203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:19.384694 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:19.384697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.385677 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.385678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:19.385806 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.386467 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:19.386477 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:19.388234 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:19.391114 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:19.391121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:19.391123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:19.391417 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:19.391739 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:19.391740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:19.391741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:19.391893 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:19.392435 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:19.392442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:19.392610 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 20:52:19.392733 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:52:19.392906 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260619 20:52:19.393205 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.379413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52071 (local address 127.0.98.62:45681)
0619 20:52:19.379641 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:19.379650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:19.379693 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:52:19.380375 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:19.380382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:19.380444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:19.380590 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:19.381596 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.383074 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.384796 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.385472 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.385573 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.388358 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:19.388363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:19.388363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:19.388374 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:19.391010 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.391517 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.391521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.391521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.391649 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.391978 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.391979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.391980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.392330 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:19.392448 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:19.392818 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 20:52:19.392960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260619 20:52:19.394703 13384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45681
I20260619 20:52:19.394809 13384 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.395031 13384 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
W20260619 20:52:19.395967 13384 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45681 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:52:19.396049 13384 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.396112 13384 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:19.396553 13384 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.396629 13384 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
W20260619 20:52:19.397069 13384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:52:19.485517 13388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:19.485733 13388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:19.485771 13388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:19.487073 13388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:19.487131 13388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:19.487176 13388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:52:19.488744 13388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38497
--tserver_master_addrs=127.0.98.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:19.488962 13388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:19.489248 13388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:19.495293 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:19.495667 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:19.496047 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:19.496569 13388 server_base.cc:1034] running on GCE node
I20260619 20:52:19.496780 13388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:19.497064 13388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:19.498250 13388 hybrid_clock.cc:648] HybridClock initialized: now 1781902339498224 us; error 38 us; skew 500 ppm
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:19.501449 13388 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:52:19.502019 13388 webserver.cc:460] Webserver started at http://127.0.98.2:38497/ using document root <none> and password file <none>
I20260619 20:52:19.502354 13388 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:19.502485 13388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:19.522195 13388 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:19.523025 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.523211 13388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:52:19.523298 13388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "5a0d12e932234cdb99f126d269a8e496"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "75b5356a1f73483056fa52e4283cde48"
server_key_iv: "f1f787e3a7acd6554f6b0913a225b4a2"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:19.523669 13388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:19.532655 13388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:19.534929 13388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:19.535101 13388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:19.535387 13388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:19.535748 13388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:19.535799 13388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.535863 13388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:19.535892 13388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.542984 13388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38643
I20260619 20:52:19.543406 13388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:52:19.543926 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38643 every 8 connection(s)
I20260619 20:52:19.549043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13388
I20260619 20:52:19.549364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39725
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45681
--webserver_port=33521
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:19.554124 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.544173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49309 (local address 127.0.98.62:45681)
0619 20:52:19.544217 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:19.544219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:19.544486 (+ 267us) server_negotiation.cc:407] Connection header received
0619 20:52:19.545001 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:19.545003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:19.545018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:19.545081 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:19.545735 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.546283 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.547401 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.547620 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.547708 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.550322 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:19.550323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:19.550324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:19.550332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:19.552335 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.552839 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.552840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.552840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.552899 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.553173 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.553174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.553175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.553362 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:19.553411 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:19.553879 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 20:52:19.553983 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:19.555032 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.544201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45681 (local address 127.0.98.2:49309)
0619 20:52:19.544358 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:19.544393 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:19.544893 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:19.545153 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:19.545158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:19.545431 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:19.545636 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:19.545641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.546392 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.546393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:19.547309 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:19.547311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.547729 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.547730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:19.547878 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.548529 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:19.548539 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:19.550210 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:19.552425 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:19.552431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:19.552434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:19.552752 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:19.552969 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:19.552969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:19.552970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:19.553104 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:19.553469 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:19.553476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:19.553704 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 20:52:19.554531 (+ 827us) client_negotiation.cc:240] Negotiation successful
0619 20:52:19.554727 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260619 20:52:19.555781 13518 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45681
I20260619 20:52:19.555881 13518 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.556097 13518 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
W20260619 20:52:19.556823 13518 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45681 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:52:19.556907 13518 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.556968 13518 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:19.557288 13518 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.557384 13518 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
W20260619 20:52:19.557725 13518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:52:19.656668 13521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:19.657032 13521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:19.657127 13521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:19.658926 13521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:19.658998 13521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:19.659052 13521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:52:19.661204 13521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33521
--tserver_master_addrs=127.0.98.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:19.661489 13521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:19.661825 13521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:19.673261 13526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:19.673261 13527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:19.674296 13521 server_base.cc:1034] running on GCE node
W20260619 20:52:19.674512 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:19.674727 13521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:19.675032 13521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:19.676283 13521 hybrid_clock.cc:648] HybridClock initialized: now 1781902339676266 us; error 34 us; skew 500 ppm
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:19.679755 13521 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:52:19.680328 13521 webserver.cc:460] Webserver started at http://127.0.98.3:33521/ using document root <none> and password file <none>
I20260619 20:52:19.680672 13521 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:19.680816 13521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:19.696128 13521 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:19.696946 13538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.697141 13521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:19.697222 13521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "bf983e24544b4d838bc31b3a05279555"
format_stamp: "Formatted at 2026-06-19 20:52:18 on dist-test-slave-0smx"
server_key: "9a93904362175b03722100a7d25b290f"
server_key_iv: "99d2b62fa14de8542e8db345349c32f7"
server_key_version: "kuduclusterkey@0"
I20260619 20:52:19.697767 13521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:19.722000 13521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:19.724573 13521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:19.724790 13521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:19.725069 13521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:19.725488 13521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:19.725570 13521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.725633 13521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:19.725677 13521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:19.731849 13521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39725
I20260619 20:52:19.731997 13651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39725 every 8 connection(s)
I20260619 20:52:19.732323 13521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:52:19.736030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13521
Jun 19 20:52:19 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:19.743784 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.733820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37063 (local address 127.0.98.62:45681)
0619 20:52:19.733908 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:19.733910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:19.734132 (+ 222us) server_negotiation.cc:407] Connection header received
0619 20:52:19.734610 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:19.734611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:19.734624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:19.734708 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:19.735310 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.735797 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.737408 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.737728 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.737789 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.740288 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:19.740290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:19.740290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:19.740297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:19.742152 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.742636 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.742637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.742637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.742692 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.743004 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.743005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.743005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.743135 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:19.743199 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:19.743596 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 20:52:19.743684 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:52:19.743975 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.733795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45681 (local address 127.0.98.3:37063)
0619 20:52:19.733973 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:19.734014 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:19.734487 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:19.734775 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:19.734778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:19.735031 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:19.735216 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:19.735220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.736249 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.736250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:19.737273 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:19.737276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.737867 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.737869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:19.738005 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.738569 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:19.738580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:19.740194 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:19.742240 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:19.742247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:19.742250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:19.742554 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:19.742782 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:19.742782 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:19.742783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:19.742928 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:19.743229 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:19.743234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:19.743413 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 20:52:19.743539 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 20:52:19.743699 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260619 20:52:19.744519 13652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45681
I20260619 20:52:19.744622 13652 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.744797 13652 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
W20260619 20:52:19.745405 13652 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45681 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:52:19.745478 13652 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.745540 13652 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:19.745700 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:19.736393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.98.62:45681)
0619 20:52:19.737450 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:19.737451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:19.737587 (+ 136us) server_negotiation.cc:407] Connection header received
0619 20:52:19.737676 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:19.737677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:19.737689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:19.737731 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:19.738757 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.739524 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.740669 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:19.740998 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:19.741061 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:19.742836 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:19.742837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:19.742837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:19.742843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:19.744535 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.744883 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.744884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.744884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.744938 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:19.745182 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:19.745183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:19.745183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:19.745327 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:19.745474 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:19.745516 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:52:19.745608 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":55,"threads_started":1}
I20260619 20:52:19.745937 13652 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:19.746009 13652 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
W20260619 20:52:19.746423 13652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 20:52:20 dist-test-slave-0smx krb5kdc[12190](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781902340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:20.397243 13384 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:20.397452 13384 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:20.397938 13185 ts_manager.cc:194] Registered new tserver with Master: 3b8ea01f0b5d4eaca171508795c1a1a8 (127.0.98.1:43283)
I20260619 20:52:20.398967 13185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52071
I20260619 20:52:20.557914 13518 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:20.558065 13518 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:20.558569 13185 ts_manager.cc:194] Registered new tserver with Master: 5a0d12e932234cdb99f126d269a8e496 (127.0.98.2:38643)
I20260619 20:52:20.559120 13185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49309
Jun 19 20:52:20 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 20:52:20.746611 13652 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:20.746779 13652 heartbeater.cc:511] Master 127.0.98.62:45681 requested a full tablet report, sending...
I20260619 20:52:20.747103 13185 ts_manager.cc:194] Registered new tserver with Master: bf983e24544b4d838bc31b3a05279555 (127.0.98.3:39725)
I20260619 20:52:20.747628 13185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37063
I20260619 20:52:20.749132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:52:20.757823 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.750210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.98.62:45681)
0619 20:52:20.750313 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:20.750316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:20.750323 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:52:20.750344 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:20.750345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:20.750356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:20.750432 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:20.751090 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.751844 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.752809 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.753128 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.753205 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.754632 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:20.754634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:20.754634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:20.754641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:20.756479 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:20.756889 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:20.756891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:20.756891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:20.756991 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:20.757200 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:20.757201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:20.757201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:20.757384 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:20.757490 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:20.757601 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 20:52:20.757687 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
I20260619 20:52:20.760226 13185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:52:20.762629 13185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:52:20.773005 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.765668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38643 (local address 127.0.0.1:35452)
0619 20:52:20.768211 (+ 2543us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:20.768221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:20.768276 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:20.768654 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:20.768657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:20.768660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:20.768731 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.768733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.770067 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.770068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.771618 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.771621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.772677 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.772678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.772765 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.772778 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:52:20.772846 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:52:20.772895 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2465,"mutex_wait_us":240,"thread_start_us":86,"threads_started":1}
I20260619 20:52:20.774142 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.765665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43283 (local address 127.0.0.1:57034)
0619 20:52:20.766277 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:20.766315 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:20.766380 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:20.768681 (+ 2301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:20.768682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:20.768684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:20.768733 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.768735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.770070 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.770071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.772615 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.772618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.773907 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.773908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.773970 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.773974 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:20.774024 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:52:20.774063 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":49,"thread_start_us":90,"threads_started":1}
I20260619 20:52:20.774438 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.765939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.0.98.1:43283)
0619 20:52:20.768462 (+ 2523us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:20.768466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:20.768475 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:52:20.768509 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:20.768511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:20.768527 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:20.768599 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:20.768807 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.769970 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.772714 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.773723 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.773788 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.773873 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:20.774318 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 20:52:20.774363 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2490,"thread_start_us":55,"threads_started":1}
I20260619 20:52:20.774993 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.766132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.98.2:38643)
0619 20:52:20.766506 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:20.766511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:20.768318 (+ 1807us) server_negotiation.cc:407] Connection header received
0619 20:52:20.768435 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:20.768439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:20.768456 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:20.768564 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:20.768814 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.769965 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.771717 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.772492 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.772568 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.772642 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:20.774865 (+ 2223us) server_negotiation.cc:299] Negotiation successful
0619 20:52:20.774914 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":52,"threads_started":1}
I20260619 20:52:20.775121 13306 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd141884bce4ff6adc0b43bf6d2fba7 (DEFAULT_TABLE table=test-table [id=a7dbec8f0d3844ac9b6c71f6c1bd3019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:20.775544 13306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd141884bce4ff6adc0b43bf6d2fba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:20.775950 13452 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd141884bce4ff6adc0b43bf6d2fba7 (DEFAULT_TABLE table=test-table [id=a7dbec8f0d3844ac9b6c71f6c1bd3019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:20.776250 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd141884bce4ff6adc0b43bf6d2fba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:20.777521 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.766014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.0.98.3:39725)
0619 20:52:20.768091 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:20.768096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:20.773656 (+ 5560us) server_negotiation.cc:407] Connection header received
0619 20:52:20.773770 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:20.773773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:20.773788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:20.773850 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:20.774085 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.775114 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.776324 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.777034 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.777085 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.777151 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:20.777391 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 20:52:20.777433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2044,"thread_start_us":49,"threads_started":1}
I20260619 20:52:20.777697 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.765930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39725 (local address 127.0.0.1:50060)
0619 20:52:20.773601 (+ 7671us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:20.773610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:20.773683 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:20.773962 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:20.773963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:20.773965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:20.773994 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.773995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.775227 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.775228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.776221 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.776224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.777209 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.777209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.777267 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:20.777270 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:52:20.777599 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 20:52:20.777638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7660,"thread_start_us":70,"threads_started":1}
I20260619 20:52:20.778419 13586 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd141884bce4ff6adc0b43bf6d2fba7 (DEFAULT_TABLE table=test-table [id=a7dbec8f0d3844ac9b6c71f6c1bd3019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:20.778717 13586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd141884bce4ff6adc0b43bf6d2fba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:20.779381 13685 tablet_bootstrap.cc:492] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: Bootstrap starting.
I20260619 20:52:20.780014 13685 tablet_bootstrap.cc:654] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:20.780314 13685 log.cc:824] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:20.780630 13686 tablet_bootstrap.cc:492] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Bootstrap starting.
I20260619 20:52:20.781219 13686 tablet_bootstrap.cc:654] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:20.781524 13686 log.cc:824] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:20.782269 13685 tablet_bootstrap.cc:492] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: No bootstrap required, opened a new log
I20260619 20:52:20.782328 13685 ts_tablet_manager.cc:1397] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:52:20.782444 13689 tablet_bootstrap.cc:492] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Bootstrap starting.
I20260619 20:52:20.783079 13689 tablet_bootstrap.cc:654] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:20.783375 13689 log.cc:824] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:20.784044 13689 tablet_bootstrap.cc:492] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: No bootstrap required, opened a new log
I20260619 20:52:20.784096 13689 ts_tablet_manager.cc:1397] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:20.785202 13685 raft_consensus.cc:348] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.785594 13685 raft_consensus.cc:374] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:20.785773 13685 raft_consensus.cc:729] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8ea01f0b5d4eaca171508795c1a1a8, State: Initialized, Role: FOLLOWER
I20260619 20:52:20.785986 13685 consensus_queue.cc:260] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.786325 13685 ts_tablet_manager.cc:1428] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 20:52:20.786496 13384 heartbeater.cc:503] Master 127.0.98.62:45681 was elected leader, sending a full tablet report...
I20260619 20:52:20.786645 13689 raft_consensus.cc:348] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.786784 13689 raft_consensus.cc:374] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:20.786819 13689 raft_consensus.cc:729] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf983e24544b4d838bc31b3a05279555, State: Initialized, Role: FOLLOWER
I20260619 20:52:20.786932 13689 consensus_queue.cc:260] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.787168 13689 ts_tablet_manager.cc:1428] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:52:20.787268 13652 heartbeater.cc:503] Master 127.0.98.62:45681 was elected leader, sending a full tablet report...
I20260619 20:52:20.788304 13686 tablet_bootstrap.cc:492] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: No bootstrap required, opened a new log
I20260619 20:52:20.788430 13686 ts_tablet_manager.cc:1397] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 20:52:20.790961 13686 raft_consensus.cc:348] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.791136 13686 raft_consensus.cc:374] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:20.791188 13686 raft_consensus.cc:729] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0d12e932234cdb99f126d269a8e496, State: Initialized, Role: FOLLOWER
I20260619 20:52:20.791304 13686 consensus_queue.cc:260] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.791561 13686 ts_tablet_manager.cc:1428] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:52:20.791682 13518 heartbeater.cc:503] Master 127.0.98.62:45681 was elected leader, sending a full tablet report...
W20260619 20:52:20.794528 13519 tablet.cc:2369] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:52:20 dist-test-slave-0smx krb5kdc[12190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 20:52:20.879885 13385 tablet.cc:2369] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:52:20.983539 13653 tablet.cc:2369] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:52:20.993722 13692 raft_consensus.cc:482] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:20.993875 13692 raft_consensus.cc:504] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:20.994256 13692 leader_election.cc:288] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a0d12e932234cdb99f126d269a8e496 (127.0.98.2:38643), 3b8ea01f0b5d4eaca171508795c1a1a8 (127.0.98.1:43283)
I20260619 20:52:21.006770 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.994656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58347 (local address 127.0.98.1:43283)
0619 20:52:20.994691 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:20.994693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:20.996714 (+ 2021us) server_negotiation.cc:407] Connection header received
0619 20:52:20.996829 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:20.996831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:20.996844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:20.996875 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:20.997132 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.997893 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.005157 (+ 7264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.005782 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.005848 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:21.005860 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:21.006648 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 20:52:21.006697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 20:52:21.007270 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.994397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43283 (local address 127.0.98.3:58347)
0619 20:52:20.996633 (+ 2236us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:20.996647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:20.996747 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:20.996947 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:20.996948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:20.996950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:20.997047 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.997049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:20.998000 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:20.998001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:20.999238 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.999241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.005923 (+ 6682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.005924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:21.005995 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:21.006000 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:52:21.007121 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0619 20:52:21.007169 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2214,"thread_start_us":98,"threads_started":1}
I20260619 20:52:21.007761 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd141884bce4ff6adc0b43bf6d2fba7" candidate_uuid: "bf983e24544b4d838bc31b3a05279555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" is_pre_election: true
I20260619 20:52:21.007929 13331 raft_consensus.cc:2454] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf983e24544b4d838bc31b3a05279555 in term 0.
I20260619 20:52:21.008260 13542 leader_election.cc:302] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8ea01f0b5d4eaca171508795c1a1a8, bf983e24544b4d838bc31b3a05279555; no voters:
I20260619 20:52:21.008405 13692 raft_consensus.cc:2781] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:52:21.008443 13692 raft_consensus.cc:482] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:52:21.008464 13692 raft_consensus.cc:3037] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:21.009418 13692 raft_consensus.cc:504] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:21.009557 13692 leader_election.cc:288] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [CANDIDATE]: Term 1 election: Requested vote from peers 5a0d12e932234cdb99f126d269a8e496 (127.0.98.2:38643), 3b8ea01f0b5d4eaca171508795c1a1a8 (127.0.98.1:43283)
I20260619 20:52:21.009886 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd141884bce4ff6adc0b43bf6d2fba7" candidate_uuid: "bf983e24544b4d838bc31b3a05279555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8"
I20260619 20:52:21.009959 13331 raft_consensus.cc:3037] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:21.010805 13331 raft_consensus.cc:2454] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf983e24544b4d838bc31b3a05279555 in term 1.
I20260619 20:52:21.011020 13542 leader_election.cc:302] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8ea01f0b5d4eaca171508795c1a1a8, bf983e24544b4d838bc31b3a05279555; no voters:
I20260619 20:52:21.011132 13692 raft_consensus.cc:2781] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:21.011278 13692 raft_consensus.cc:686] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 1 LEADER]: Becoming Leader. State: Replica: bf983e24544b4d838bc31b3a05279555, State: Running, Role: LEADER
I20260619 20:52:21.011384 13692 consensus_queue.cc:237] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } }
I20260619 20:52:21.012858 13186 catalog_manager.cc:5495] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf983e24544b4d838bc31b3a05279555 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bf983e24544b4d838bc31b3a05279555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf983e24544b4d838bc31b3a05279555" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39725 } health_report { overall_health: HEALTHY } } }
I20260619 20:52:21.014256 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.994610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53623 (local address 127.0.98.2:38643)
0619 20:52:20.994655 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:20.994658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:20.999430 (+ 4772us) server_negotiation.cc:407] Connection header received
0619 20:52:20.999534 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:20.999536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:20.999551 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:20.999581 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:20.999781 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.000458 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.012225 (+ 11767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.012659 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.012700 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:21.012707 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:21.014131 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0619 20:52:21.014185 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:52:21.014647 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:20.994397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38643 (local address 127.0.98.3:53623)
0619 20:52:20.999368 (+ 4971us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:20.999379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:20.999456 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:20.999645 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:20.999645 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:20.999647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:20.999708 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:20.999709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.004381 (+ 4672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.004383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:21.006540 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:21.006543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.013960 (+ 7417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.013962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:21.014039 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:21.014043 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:21.014520 (+ 477us) client_negotiation.cc:240] Negotiation successful
0619 20:52:21.014569 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4953,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260619 20:52:21.015095 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd141884bce4ff6adc0b43bf6d2fba7" candidate_uuid: "bf983e24544b4d838bc31b3a05279555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0d12e932234cdb99f126d269a8e496" is_pre_election: true
I20260619 20:52:21.015234 13472 raft_consensus.cc:2454] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf983e24544b4d838bc31b3a05279555 in term 0.
I20260619 20:52:21.015352 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd141884bce4ff6adc0b43bf6d2fba7" candidate_uuid: "bf983e24544b4d838bc31b3a05279555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0d12e932234cdb99f126d269a8e496"
I20260619 20:52:21.015403 13471 raft_consensus.cc:3037] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:21.016263 13471 raft_consensus.cc:2454] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf983e24544b4d838bc31b3a05279555 in term 1.
I20260619 20:52:21.060926 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:21.057537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.0.98.3:39725)
0619 20:52:21.057583 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:21.057585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:21.057595 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:52:21.057639 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:21.057641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:21.057666 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:21.057759 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:52:21.057954 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.058707 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.059830 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:21.060135 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:21.060190 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:21.060382 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:52:21.060517 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:52:21.060590 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:21.060821 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 20:52:21.060844 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:52:21.063736 13331 raft_consensus.cc:1261] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 3b8ea01f0b5d4eaca171508795c1a1a8 [term 1 FOLLOWER]: Refusing update from remote peer bf983e24544b4d838bc31b3a05279555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:21.064134 13698 consensus_queue.cc:1035] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8ea01f0b5d4eaca171508795c1a1a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:21.066849 13471 raft_consensus.cc:1261] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 1 FOLLOWER]: Refusing update from remote peer bf983e24544b4d838bc31b3a05279555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:21.067183 13698 consensus_queue.cc:1035] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0d12e932234cdb99f126d269a8e496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:21.071622 13701 mvcc.cc:205] Tried to move back new op lower bound from 7298671988994408448 to 7298671988783153152. Current Snapshot: MvccSnapshot[applied={T|T < 7298671988994408448}]
I20260619 20:52:21.071771 13704 mvcc.cc:205] Tried to move back new op lower bound from 7298671988994408448 to 7298671988783153152. Current Snapshot: MvccSnapshot[applied={T|T < 7298671988994408448}]
I20260619 20:52:21.077186 13186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:21.077280 13186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:21.078279 13186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ac1ed225f8844a5be4d7a0d7b6f7eee: Sending DeleteTablet for 3 replicas of tablet 1dd141884bce4ff6adc0b43bf6d2fba7
I20260619 20:52:21.078694 13452 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd141884bce4ff6adc0b43bf6d2fba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260619 20:52:21.079072 13586 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd141884bce4ff6adc0b43bf6d2fba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50060
I20260619 20:52:21.079229 13306 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd141884bce4ff6adc0b43bf6d2fba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57034
I20260619 20:52:21.079349 13710 tablet_replica.cc:330] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: stopping tablet replica
I20260619 20:52:21.079440 13710 raft_consensus.cc:2229] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:21.079533 13710 raft_consensus.cc:2258] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:21.079631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13224
I20260619 20:52:21.079988 13710 ts_tablet_manager.cc:1905] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:21.081596 13710 ts_tablet_manager.cc:1918] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:21.081658 13710 log.cc:1196] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/1dd141884bce4ff6adc0b43bf6d2fba7
I20260619 20:52:21.081941 13710 ts_tablet_manager.cc:1939] T 1dd141884bce4ff6adc0b43bf6d2fba7 P 5a0d12e932234cdb99f126d269a8e496: Deleting consensus metadata
I20260619 20:52:21.082335 13170 catalog_manager.cc:4841] TS 5a0d12e932234cdb99f126d269a8e496 (127.0.98.2:38643): tablet 1dd141884bce4ff6adc0b43bf6d2fba7 (table test-table [id=a7dbec8f0d3844ac9b6c71f6c1bd3019]) successfully deleted
I20260619 20:52:21.089411 13711 tablet_replica.cc:330] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: stopping tablet replica
I20260619 20:52:21.089604 13711 raft_consensus.cc:2229] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:52:21.089792 13711 raft_consensus.cc:2258] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:21.090323 13711 ts_tablet_manager.cc:1905] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:21.091934 13711 ts_tablet_manager.cc:1918] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:21.092056 13711 log.cc:1196] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/1dd141884bce4ff6adc0b43bf6d2fba7
I20260619 20:52:21.092388 13711 ts_tablet_manager.cc:1939] T 1dd141884bce4ff6adc0b43bf6d2fba7 P bf983e24544b4d838bc31b3a05279555: Deleting consensus metadata
I20260619 20:52:21.092852 13172 catalog_manager.cc:4841] TS bf983e24544b4d838bc31b3a05279555 (127.0.98.3:39725): tablet 1dd141884bce4ff6adc0b43bf6d2fba7 (table test-table [id=a7dbec8f0d3844ac9b6c71f6c1bd3019]) successfully deleted
W20260619 20:52:21.100981 13173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43283 (error 108)
I20260619 20:52:21.102167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13388
I20260619 20:52:21.109797 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:21.109630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43283 (local address 127.0.0.1:57044)
0619 20:52:21.109697 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:21.109749 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:52:21.110162 13173 catalog_manager.cc:4584] TS 3b8ea01f0b5d4eaca171508795c1a1a8 (127.0.98.1:43283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1dd141884bce4ff6adc0b43bf6d2fba7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43283: connect: Connection refused (error 111)
I20260619 20:52:21.110796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13521
I20260619 20:52:21.126368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13153
I20260619 20:52:21.132438 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 20:52:21.477154 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 20:52:21.477273 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 20:52:21.821568 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 20:52:21.821 UTC [12220] LOG: received smart shutdown request
2026-06-19 20:52:21.825 UTC [12220] LOG: background worker "logical replication launcher" (PID 12229) exited with exit code 1
2026-06-19 20:52:21.825 UTC [12224] LOG: shutting down
2026-06-19 20:52:21.978 UTC [12220] LOG: database system is shut down
2026-06-19T20:52:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46400 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:52:22 dist-test-slave-0smx krb5kdc[13725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:52:22 dist-test-slave-0smx krb5kdc[13725](info): set up 2 sockets
Jun 19 20:52:22 dist-test-slave-0smx krb5kdc[13725](info): commencing operation
krb5kdc: starting...
W20260619 20:52:23.113204 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:52:23.227886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42591,127.0.98.61:37611,127.0.98.60:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:23.306758 13738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:23.306979 13738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:23.307003 13738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:23.308101 13738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:23.308136 13738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:23.308151 13738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:23.308168 13738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:23.308180 13738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:23.309791 13738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42591,127.0.98.61:37611,127.0.98.60:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:23.310032 13738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:23.310329 13738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:23.315460 13746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:23.315461 13738 server_base.cc:1034] running on GCE node
W20260619 20:52:23.315431 13744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:23.315433 13743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:23.316123 13738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:23.316425 13738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:23.317606 13738 hybrid_clock.cc:648] HybridClock initialized: now 1781902343317597 us; error 40 us; skew 500 ppm
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:23.322806 13738 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:23.323277 13738 webserver.cc:460] Webserver started at http://127.0.98.62:45725/ using document root <none> and password file <none>
I20260619 20:52:23.323516 13738 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:23.323593 13738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:23.323745 13738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:23.324815 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "a22c074a128b4178a54666ab07697600"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "6edb6e7f2a2a640f1b60e24af5571927"
server_key_iv: "befc55f95a8940ecae4c92d68817445a"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.325179 13738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "a22c074a128b4178a54666ab07697600"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "6edb6e7f2a2a640f1b60e24af5571927"
server_key_iv: "befc55f95a8940ecae4c92d68817445a"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.326535 13738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:52:23.327301 13753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:23.327471 13738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:52:23.327596 13738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "a22c074a128b4178a54666ab07697600"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "6edb6e7f2a2a640f1b60e24af5571927"
server_key_iv: "befc55f95a8940ecae4c92d68817445a"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.327673 13738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:23.337852 13738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:23.340060 13738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:23.340217 13738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:23.344235 13738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42591
I20260619 20:52:23.344265 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42591 every 8 connection(s)
I20260619 20:52:23.344604 13738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:23.345500 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:23.347000 13806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:52:23.352393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13738
I20260619 20:52:23.352481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-0/wal/instance
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:23.352671 392 external_mini_cluster.cc:1428] Setting key 44f1445500004e25314ac860df7d330d
I20260619 20:52:23.356350 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.347756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.98.62:42591)
0619 20:52:23.347931 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.347935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.347962 (+ 27us) server_negotiation.cc:407] Connection header received
0619 20:52:23.348429 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.348431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.348447 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.348527 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.349031 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.349889 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.350720 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.351023 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.351101 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.353123 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.353129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.353130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.353136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.354753 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.355163 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.355166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.355166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.355245 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.355467 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.355468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.355469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.355744 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.355836 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.356012 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.356131 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260619 20:52:23.356350 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.347650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42591 (local address 127.0.0.1:34618)
0619 20:52:23.347830 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.347871 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.348300 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.348552 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.348554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.348796 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.348948 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.348952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.349971 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.349972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.350630 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.350632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.351091 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.351092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.351146 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.351630 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.351639 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.353010 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.354868 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.354874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.354876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.355068 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.355313 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.355314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.355314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.355402 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.355814 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.355819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.355879 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.355989 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.356184 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":72,"thread_start_us":110,"threads_started":1}
I20260619 20:52:23.357084 13806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:52:23.357487 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.357369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44710)
0619 20:52:23.357406 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.357458 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260619 20:52:23.357620 13755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
I20260619 20:52:23.358043 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.357935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44726)
0619 20:52:23.357990 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.358022 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:52:23.358188 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37611: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 20:52:23.375649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37611
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42591,127.0.98.61:37611,127.0.98.60:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:23.407416 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } attempt: 1
I20260619 20:52:23.407928 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.407782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44728)
0619 20:52:23.407844 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.407892 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:52:23.408306 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.408219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44732)
0619 20:52:23.408263 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.408289 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:52:23.408468 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37611: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
W20260619 20:52:23.454989 13813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:23.455188 13813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:23.455209 13813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:23.456341 13813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:23.456384 13813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:23.456398 13813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:23.456411 13813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:23.456423 13813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:23.457908 13813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42591,127.0.98.61:37611,127.0.98.60:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:23.458142 13813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:23.458377 13813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:23.463114 13819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:23.463137 13821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:23.463177 13818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:23.463343 13813 server_base.cc:1034] running on GCE node
I20260619 20:52:23.463558 13813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:23.463907 13813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:23.465049 13813 hybrid_clock.cc:648] HybridClock initialized: now 1781902343465040 us; error 26 us; skew 500 ppm
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:23.467562 13813 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 20:52:23.467973 13813 webserver.cc:460] Webserver started at http://127.0.98.61:34435/ using document root <none> and password file <none>
I20260619 20:52:23.468190 13813 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:23.468235 13813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:23.468410 13813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:23.469552 13813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data/instance:
uuid: "4677934ebf2d405e8b5aa173cf84a136"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "50f8cb04e70e14d2391118fbfdb3380f"
server_key_iv: "e601c4eb7119df4f96adafac4156c97b"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.469933 13813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal/instance:
uuid: "4677934ebf2d405e8b5aa173cf84a136"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "50f8cb04e70e14d2391118fbfdb3380f"
server_key_iv: "e601c4eb7119df4f96adafac4156c97b"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.471217 13813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:23.471909 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:23.472087 13813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:23.472175 13813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal
uuid: "4677934ebf2d405e8b5aa173cf84a136"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "50f8cb04e70e14d2391118fbfdb3380f"
server_key_iv: "e601c4eb7119df4f96adafac4156c97b"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.472250 13813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:23.476686 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } attempt: 2
I20260619 20:52:23.477078 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.476938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44744)
0619 20:52:23.476994 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.477023 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:52:23.477417 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.477345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44756)
0619 20:52:23.477373 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.477396 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260619 20:52:23.477506 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37611: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37611: connect: Connection refused (error 111)
I20260619 20:52:23.483969 13813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:23.486280 13813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:23.486434 13813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:23.490175 13813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37611
I20260619 20:52:23.490206 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37611 every 8 connection(s)
I20260619 20:52:23.490525 13813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:52:23.491322 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:23.492772 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:23.500021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13813
I20260619 20:52:23.500126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-1/wal/instance
I20260619 20:52:23.500329 392 external_mini_cluster.cc:1428] Setting key 7ad2e12ecd243ef8133b32d1d7991225
I20260619 20:52:23.501112 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.493467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42591 (local address 127.0.0.1:34634)
0619 20:52:23.493629 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.493663 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.494077 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.494284 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.494287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.494522 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.494726 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.494730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.495357 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.495358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.496129 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.496131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.496444 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.496445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.496536 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.496946 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.496953 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.498313 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.499826 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.499830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.499832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.500021 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.500229 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.500230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.500230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.500331 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.500674 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.500679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.500780 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.500865 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.500969 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260619 20:52:23.501137 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.493472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.0.98.62:42591)
0619 20:52:23.493517 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.493520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.493754 (+ 234us) server_negotiation.cc:407] Connection header received
0619 20:52:23.494151 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.494153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.494162 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.494215 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.494795 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.495283 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.496196 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.496379 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.496428 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.498392 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.498394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.498394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.498400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.499759 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.500093 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.500095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.500095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.500158 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.500444 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.500445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.500446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.500580 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.500668 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.500910 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.500999 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:52:23.502040 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 20:52:23.509490 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.502287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44760)
0619 20:52:23.502332 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.502342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.502381 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.502671 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.502672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.502780 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.502823 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.502824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.503656 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.503657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.504287 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.504289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.504764 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.504765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.504830 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.505203 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.505208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.506627 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.508506 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.508507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.508507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.508656 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.508836 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.508837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.508837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.508883 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.509220 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.509221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.509254 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.509369 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.509432 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:52:23.509604 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.502368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.0.98.61:37611)
0619 20:52:23.502522 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.502527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.502547 (+ 20us) server_negotiation.cc:407] Connection header received
0619 20:52:23.502568 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.502570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.502583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.502663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.502902 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.503566 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.504388 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.504672 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.504766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.506739 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.506749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.506749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.506756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.508432 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.508727 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.508729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.508729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.508778 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.508947 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.508947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.508948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.509153 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.509236 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.509405 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.509486 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260619 20:52:23.510142 13881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 20:52:23.510601 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.510477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58686)
0619 20:52:23.510521 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.510571 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:52:23.510715 13830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
I20260619 20:52:23.511195 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.511096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58702)
0619 20:52:23.511143 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.511179 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:52:23.511287 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 20:52:23.524048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46599
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42591,127.0.98.61:37611,127.0.98.60:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:23.560531 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } attempt: 1
I20260619 20:52:23.561103 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.560920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58712)
0619 20:52:23.560995 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.561059 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 20:52:23.561534 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.561447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58722)
0619 20:52:23.561478 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.561511 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 20:52:23.561635 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
I20260619 20:52:23.568645 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } attempt: 3
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:52:23.575982 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.569040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.61:37611)
0619 20:52:23.569103 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.569105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.569116 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:52:23.569164 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.569165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.569174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.569252 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.569667 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.570191 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.570878 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.571096 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.571175 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.573416 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.573417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.573417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.573424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.574849 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.575247 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.575247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.575248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.575321 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.575527 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.575528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.575528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.575633 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.575682 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.575861 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.575923 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:52:23.576013 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.568961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44762)
0619 20:52:23.569020 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.569034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.569103 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.569258 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.569259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.569460 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.569525 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.569527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.570282 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.570284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.570772 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.570774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.571172 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.571173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.571253 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.571814 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.571819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.573327 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.574923 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.574924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.574925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.575173 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.575400 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.575400 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.575401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.575468 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.575692 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.575693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.575741 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.575826 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.575922 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:52:23.576380 13806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:52:23.576718 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.576606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58732)
0619 20:52:23.576663 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.576694 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:52:23.577008 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.576952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58738)
0619 20:52:23.576975 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.576995 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 20:52:23.577100 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
W20260619 20:52:23.606153 13888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:23.606341 13888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:23.606361 13888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:23.607443 13888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:23.607481 13888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:23.607496 13888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:23.607507 13888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:23.607556 13888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:23.608222 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } attempt: 1
I20260619 20:52:23.608701 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.608574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58746)
0619 20:52:23.608634 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.608673 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:52:23.609180 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.609053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58756)
0619 20:52:23.609101 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.609151 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:52:23.609280 13806 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
I20260619 20:52:23.609473 13888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42591,127.0.98.61:37611,127.0.98.60:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:23.609720 13888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:23.609951 13888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:23.614568 13894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:23.614614 13896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:23.614629 13888 server_base.cc:1034] running on GCE node
W20260619 20:52:23.614586 13893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:23.615000 13888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:23.615316 13888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:23.616490 13888 hybrid_clock.cc:648] HybridClock initialized: now 1781902343616476 us; error 41 us; skew 500 ppm
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:23.619287 13888 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 20:52:23.619719 13888 webserver.cc:460] Webserver started at http://127.0.98.60:36727/ using document root <none> and password file <none>
I20260619 20:52:23.619935 13888 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:23.619999 13888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:23.620139 13888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:23.621174 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data/instance:
uuid: "f857b34dbbad4a8d9b716641c98bd8b6"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "efafa98293e3e143798a136daaa9263a"
server_key_iv: "a30e02496014d83a6f6184e2ae701ae7"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.621569 13888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal/instance:
uuid: "f857b34dbbad4a8d9b716641c98bd8b6"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "efafa98293e3e143798a136daaa9263a"
server_key_iv: "a30e02496014d83a6f6184e2ae701ae7"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.622845 13888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:23.623525 13903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:23.623700 13888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:23.623801 13888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal
uuid: "f857b34dbbad4a8d9b716641c98bd8b6"
format_stamp: "Formatted at 2026-06-19 20:52:23 on dist-test-slave-0smx"
server_key: "efafa98293e3e143798a136daaa9263a"
server_key_iv: "a30e02496014d83a6f6184e2ae701ae7"
server_key_version: "encryptionkey@0"
I20260619 20:52:23.623871 13888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:23.629760 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } attempt: 2
I20260619 20:52:23.630108 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.629996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58764)
0619 20:52:23.630049 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.630084 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 20:52:23.630427 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.630326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58770)
0619 20:52:23.630383 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.630411 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 20:52:23.630507 13881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46599: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
I20260619 20:52:23.678893 13888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:23.680790 13888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:23.680979 13888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:23.684496 13806 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } attempt: 2
I20260619 20:52:23.684953 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.684815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58784)
0619 20:52:23.684878 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.684912 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 20:52:23.685144 13888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46599
I20260619 20:52:23.685186 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46599 every 8 connection(s)
I20260619 20:52:23.685617 13888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:52:23.686347 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:23.687863 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:52:23.689309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 13888
I20260619 20:52:23.689410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781902166366061-392-0/minicluster-data/master-2/wal/instance
I20260619 20:52:23.689602 392 external_mini_cluster.cc:1428] Setting key c58583a8b9c9cb6953a0394780830c10
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:52:23.695371 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.685265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58788)
0619 20:52:23.685328 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.685338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.685380 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.686815 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.686816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.686986 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.687035 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.687036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.688130 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.688131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.688830 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.688833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.689623 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.689624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.689685 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.690120 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.690126 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.691524 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.693803 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.693805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.693805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.694024 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.694313 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.694314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.694315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.694383 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.694995 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.694996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.695045 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.695124 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.695257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 20:52:23.695583 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.685472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.0.98.60:46599)
0619 20:52:23.686072 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.686080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.686103 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:52:23.686620 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.686627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.686674 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.686822 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.687131 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.688010 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.688929 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.689524 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.689658 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.691626 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.691630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.691630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.691641 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.693719 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.694122 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.694126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.694126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.694232 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.694566 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.694567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.694567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.694890 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.694965 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.695215 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.695347 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":481,"threads_started":1}
I20260619 20:52:23.697057 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.688468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42591 (local address 127.0.0.1:34650)
0619 20:52:23.688622 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.688664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.688752 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.688904 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.688907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.689244 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.689353 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.689355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.690265 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.690266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.691197 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.691200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.691624 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.691625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.691698 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.692391 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.692396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.694002 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.695821 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.695827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.695829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.696084 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.696307 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.696308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.696308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.696373 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.696708 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.696714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.696761 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.696853 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.696959 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260619 20:52:23.697103 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.688521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.98.62:42591)
0619 20:52:23.688561 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.688565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.688734 (+ 169us) server_negotiation.cc:407] Connection header received
0619 20:52:23.688799 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.688800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.688811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.688883 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.689458 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.690143 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.691306 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.691499 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.691530 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.694097 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.694098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.694099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.694106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.695716 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.696158 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.696159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.696159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.696212 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.696447 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.696448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.696449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.696633 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.696668 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.696892 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.696988 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:23.697594 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: Bootstrap starting.
I20260619 20:52:23.698634 13806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:23.698964 13806 log.cc:824] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:23.699261 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:52:23.699664 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.690129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.98.60:46599)
0619 20:52:23.690250 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.690252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.690260 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:52:23.690280 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.690281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.690291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.690365 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.691065 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.691738 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.692720 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.693031 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.693132 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.695855 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.695857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.695857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.695863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.698454 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.698851 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.698852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.698852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.698914 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.699123 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.699124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.699125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.699238 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.699427 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.699464 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.699537 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260619 20:52:23.699998 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: No bootstrap required, opened a new log
I20260619 20:52:23.701618 13806 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.701719 13806 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:23.701740 13806 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22c074a128b4178a54666ab07697600, State: Initialized, Role: FOLLOWER
I20260619 20:52:23.701851 13806 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.702126 13806 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 20:52:23.703939 13974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260619 20:52:23.703989 13974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:52:23.704589 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.704679 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:23.705308 13738 master_runner.cc:186] Error getting master registration for 127.0.98.61:37611: OK, instance_id { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" instance_seqno: 1781902343488964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:52:23.708024 13738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:52:23.708278 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.699615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.0.98.61:37611)
0619 20:52:23.699700 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.699702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.699911 (+ 209us) server_negotiation.cc:407] Connection header received
0619 20:52:23.699957 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.699958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.699968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.700015 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.700342 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.700811 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.702017 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.702308 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.702382 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.704756 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.704758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.704759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.704766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.706476 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.706906 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.706907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.706907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.706966 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.707436 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.707437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.707437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.707604 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.707680 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.708106 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.708187 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 20:52:23.708575 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.699495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37611 (local address 127.0.0.1:44764)
0619 20:52:23.699839 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.699849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.699897 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.700015 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.700015 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.700185 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.700247 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.700250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.701113 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.701114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.701925 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.701927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.702434 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.702435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.702503 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.702917 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.702922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.704654 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.706561 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.706562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.706562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.706782 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.707036 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.707037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.707037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.707104 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.707918 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.707919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.707963 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.708400 (+ 437us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.708490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328}
I20260619 20:52:23.708876 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:52:23.709833 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.700892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.0.98.62:42591)
0619 20:52:23.700953 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.700954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.701001 (+ 47us) server_negotiation.cc:407] Connection header received
0619 20:52:23.701063 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.701064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.701073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.701121 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.701717 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.702267 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.703626 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.703945 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.703985 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.706914 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.706915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.706915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.706921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.708495 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.708929 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.708930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.708930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.708989 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.709203 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.709204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.709205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.709361 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.709466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.709663 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.709738 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:52:23.716750 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.709140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.0.98.60:46599)
0619 20:52:23.709179 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.709181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.709248 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:52:23.709342 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.709342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.709352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.709408 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.709667 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.710380 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.711297 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.711602 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.711670 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.713998 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.713999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.713999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.714005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.715637 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.715983 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.715984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.715985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.716049 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.716257 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.716257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.716258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.716371 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.716409 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.716573 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.716645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:23.717012 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.709095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58804)
0619 20:52:23.709206 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.709215 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.709260 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.709409 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.709410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.709546 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.709576 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.709577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.710465 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.710465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.711217 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.711219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.711813 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.711814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.711870 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.712271 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.712276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.713909 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.715730 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.715730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.715731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.715920 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.716118 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.716119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.716120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.716180 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.716447 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.716448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.716479 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.716855 (+ 376us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.716931 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260619 20:52:23.717788 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.710369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.0.98.61:37611)
0619 20:52:23.710403 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.710404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.710412 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:52:23.710470 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.710471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.710479 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.710533 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.711109 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.711784 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.712446 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.712650 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.712724 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.715100 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.715102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.715102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.715109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.716567 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.716974 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.716975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.716975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.717028 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.717222 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.717223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.717223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.717380 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.717560 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.717627 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.717703 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:23.718441 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6: Bootstrap starting.
I20260619 20:52:23.719079 13956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:23.719396 13956 log.cc:824] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:23.720165 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6: No bootstrap required, opened a new log
I20260619 20:52:23.721180 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:52:23.721864 13956 raft_consensus.cc:348] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.722009 13956 raft_consensus.cc:374] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:23.722064 13956 raft_consensus.cc:729] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f857b34dbbad4a8d9b716641c98bd8b6, State: Initialized, Role: FOLLOWER
I20260619 20:52:23.722218 13956 consensus_queue.cc:260] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.722486 13881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } attempt: 3
I20260619 20:52:23.722553 13956 sys_catalog.cc:564] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:23.723400 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.723559 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:52:23.724647 13996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:52:23.724743 13996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 20:52:23 dist-test-slave-0smx krb5kdc[13725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:52:23.725751 13888 master_runner.cc:186] Error getting master registration for 127.0.98.61:37611: OK, instance_id { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" instance_seqno: 1781902343488964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:52:23.728217 13888 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:52:23.730235 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.722692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46599 (local address 127.0.0.1:58820)
0619 20:52:23.722765 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:23.722775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:23.722809 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:23.723012 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:23.723013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:23.723169 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:23.723208 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.723210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.724170 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.724171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.724995 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:23.724997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.725507 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.725508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:23.725561 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.725994 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:23.725999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:23.727447 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:23.729184 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.729185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.729186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.729433 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.729627 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:23.729628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:23.729628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:23.729697 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:23.729948 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:23.729949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:23.729988 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:52:23.730047 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:52:23.730137 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 20:52:23.730896 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.722872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.0.98.60:46599)
0619 20:52:23.722902 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.722904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.722910 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:52:23.722927 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.722928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.722936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.722979 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.723284 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.724027 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.725085 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.725402 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.725459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.727538 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.727539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.727540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.727546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.729099 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.729512 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.729513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.729513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.729565 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.729760 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.729761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.729761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.729872 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.729925 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.730738 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.730812 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:52:23.732436 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: Bootstrap starting.
I20260619 20:52:23.733121 13881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:23.733527 13881 log.cc:824] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:23.734195 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.723264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.61:37611)
0619 20:52:23.723343 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.723345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.725527 (+ 2182us) server_negotiation.cc:407] Connection header received
0619 20:52:23.725636 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.725637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.725647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.725677 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.726939 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.727631 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.729376 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.729716 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.729764 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.731316 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.731317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.731318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.731324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.732964 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.733378 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.733379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.733380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.733431 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.733633 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.733634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.733634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.733751 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.733836 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.734040 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.734113 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 20:52:23.737928 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: No bootstrap required, opened a new log
I20260619 20:52:23.739763 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.724129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.0.98.60:46599)
0619 20:52:23.724181 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.724183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.728288 (+ 4105us) server_negotiation.cc:407] Connection header received
0619 20:52:23.728309 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.728309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.728318 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.728374 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.730059 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.730839 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.733663 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.733973 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.734012 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.736157 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.736158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.736159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.736166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.737735 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.738676 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.738678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.738678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.738729 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.738916 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.738917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.738917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.739029 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.739120 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.739600 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.739679 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:52:23.740656 13881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.740793 13881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:23.740826 13881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4677934ebf2d405e8b5aa173cf84a136, State: Initialized, Role: FOLLOWER
I20260619 20:52:23.740931 13881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.741212 13881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:23.742756 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:23.723176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.0.98.62:42591)
0619 20:52:23.723228 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:23.723230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:23.728462 (+ 5232us) server_negotiation.cc:407] Connection header received
0619 20:52:23.729222 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:23.729223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:23.729233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:23.729261 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:23.731860 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.732632 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.734622 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:23.734899 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:23.734939 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:23.739295 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:23.739296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:23.739297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:23.739304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:23.741457 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.741851 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.741852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.741853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.741908 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:23.742116 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:23.742117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:23.742118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:23.742242 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:23.742327 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:23.742566 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:52:23.742658 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:52:23.745242 13813 master_runner.cc:263] No leader master found from master 4677934ebf2d405e8b5aa173cf84a136
I20260619 20:52:23.745324 13813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:52:23.745368 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.745494 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:52:23.750041 14010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:52:23.750134 14010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 20:52:23.840960 13759 tablet.cc:2369] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:52:23.856285 13982 raft_consensus.cc:482] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:23.856418 13982 raft_consensus.cc:504] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.856665 13982 leader_election.cc:288] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22c074a128b4178a54666ab07697600 (127.0.98.62:42591), 4677934ebf2d405e8b5aa173cf84a136 (127.0.98.61:37611)
I20260619 20:52:23.856905 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22c074a128b4178a54666ab07697600" is_pre_election: true
I20260619 20:52:23.857061 13780 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f857b34dbbad4a8d9b716641c98bd8b6 in term 0.
I20260619 20:52:23.857024 13855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4677934ebf2d405e8b5aa173cf84a136" is_pre_election: true
I20260619 20:52:23.857172 13855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f857b34dbbad4a8d9b716641c98bd8b6 in term 0.
I20260619 20:52:23.857376 13905 leader_election.cc:302] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4677934ebf2d405e8b5aa173cf84a136, f857b34dbbad4a8d9b716641c98bd8b6; no voters:
I20260619 20:52:23.857543 13982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:52:23.857614 13982 raft_consensus.cc:482] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:52:23.857649 13982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:23.858356 13982 raft_consensus.cc:504] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.858480 13982 leader_election.cc:288] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [CANDIDATE]: Term 1 election: Requested vote from peers a22c074a128b4178a54666ab07697600 (127.0.98.62:42591), 4677934ebf2d405e8b5aa173cf84a136 (127.0.98.61:37611)
I20260619 20:52:23.858675 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22c074a128b4178a54666ab07697600"
I20260619 20:52:23.858701 13855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4677934ebf2d405e8b5aa173cf84a136"
I20260619 20:52:23.858745 13780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:23.858759 13855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:23.859556 13855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f857b34dbbad4a8d9b716641c98bd8b6 in term 1.
I20260619 20:52:23.859550 13780 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f857b34dbbad4a8d9b716641c98bd8b6 in term 1.
I20260619 20:52:23.859719 13905 leader_election.cc:302] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22c074a128b4178a54666ab07697600, f857b34dbbad4a8d9b716641c98bd8b6; no voters:
I20260619 20:52:23.859830 13982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:23.859969 13982 raft_consensus.cc:686] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [term 1 LEADER]: Becoming Leader. State: Replica: f857b34dbbad4a8d9b716641c98bd8b6, State: Running, Role: LEADER
I20260619 20:52:23.860062 13982 consensus_queue.cc:237] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } }
I20260619 20:52:23.860415 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f857b34dbbad4a8d9b716641c98bd8b6. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.860546 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:23.860765 14014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:23.860965 14014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:23.864014 13780 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [term 1 FOLLOWER]: Refusing update from remote peer f857b34dbbad4a8d9b716641c98bd8b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:23.864319 13855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [term 1 FOLLOWER]: Refusing update from remote peer f857b34dbbad4a8d9b716641c98bd8b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:23.864311 13982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:23.864652 13982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:23.865513 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f857b34dbbad4a8d9b716641c98bd8b6. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.865594 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:52:23.866397 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f857b34dbbad4a8d9b716641c98bd8b6. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.866496 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:52:23.867352 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.867426 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:23.867735 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.867825 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:23.867868 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.867936 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:52:23.868193 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c074a128b4178a54666ab07697600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42591 } } peers { permanent_uuid: "4677934ebf2d405e8b5aa173cf84a136" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37611 } } peers { permanent_uuid: "f857b34dbbad4a8d9b716641c98bd8b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46599 } } }
I20260619 20:52:23.868216 14016 mvcc.cc:205] Tried to move back new op lower bound from 7298672000464953344 to 7298672000451416064. Current Snapshot: MvccSnapshot[applied={T|T < 7298672000464953344}]
I20260619 20:52:23.868266 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:52:23.868500 14018 mvcc.cc:205] Tried to move back new op lower bound from 7298672000464953344 to 7298672000451416064. Current Snapshot: MvccSnapshot[applied={T|T < 7298672000464953344}]
I20260619 20:52:23.869212 14014 catalog_manager.cc:1347] Generated new cluster ID: df48d9854afd40049b058efcca65487d
I20260619 20:52:23.869280 14014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:52:23.879647 14014 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:52:23.880385 14014 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:52:23.890309 14014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f857b34dbbad4a8d9b716641c98bd8b6: Generated new TSK 0
I20260619 20:52:23.890493 14014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:52:24.704386 13974 catalog_manager.cc:1259] Loaded cluster ID: df48d9854afd40049b058efcca65487d
I20260619 20:52:24.704463 13974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: loading cluster ID for follower catalog manager: success
I20260619 20:52:24.705284 13974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: acquiring CA information for follower catalog manager: success
I20260619 20:52:24.705634 13974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a22c074a128b4178a54666ab07697600: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:52:24.750630 14010 catalog_manager.cc:1259] Loaded cluster ID: df48d9854afd40049b058efcca65487d
I20260619 20:52:24.750705 14010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: loading cluster ID for follower catalog manager: success
I20260619 20:52:24.751523 14010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: acquiring CA information for follower catalog manager: success
I20260619 20:52:24.751828 14010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4677934ebf2d405e8b5aa173cf84a136: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:52:24.942612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13738
I20260619 20:52:24.947798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13813
I20260619 20:52:24.952512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 13888
2026-06-19T20:52:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2862 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:52:25 dist-test-slave-0smx krb5kdc[14030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:52:25 dist-test-slave-0smx krb5kdc[14030](info): set up 2 sockets
Jun 19 20:52:25 dist-test-slave-0smx krb5kdc[14030](info): commencing operation
krb5kdc: starting...
W20260619 20:52:25.989279 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:52:26 dist-test-slave-0smx krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:52:26.104557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38699
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:26.186265 14043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:26.186488 14043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:26.186513 14043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:26.187582 14043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:26.187620 14043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:26.187636 14043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:26.187650 14043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:26.187664 14043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:26.189054 14043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:26.189244 14043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:26.189443 14043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:26.194061 14051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:26.194042 14048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:26.194314 14043 server_base.cc:1034] running on GCE node
W20260619 20:52:26.194069 14049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:26.194600 14043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:26.194877 14043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:26.196025 14043 hybrid_clock.cc:648] HybridClock initialized: now 1781902346196011 us; error 30 us; skew 500 ppm
Jun 19 20:52:26 dist-test-slave-0smx krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:26.198419 14043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:26.198891 14043 webserver.cc:460] Webserver started at http://127.0.98.62:40597/ using document root <none> and password file <none>
I20260619 20:52:26.199118 14043 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:26.199185 14043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:26.199339 14043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:26.200332 14043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "30dc00e2e47a430b9d794d1897f3d985"
format_stamp: "Formatted at 2026-06-19 20:52:26 on dist-test-slave-0smx"
server_key: "184c5f0479b72463856fea536132ec68"
server_key_iv: "b5ce3b0a7410a901964fc8cb4845215c"
server_key_version: "encryptionkey@0"
I20260619 20:52:26.200685 14043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "30dc00e2e47a430b9d794d1897f3d985"
format_stamp: "Formatted at 2026-06-19 20:52:26 on dist-test-slave-0smx"
server_key: "184c5f0479b72463856fea536132ec68"
server_key_iv: "b5ce3b0a7410a901964fc8cb4845215c"
server_key_version: "encryptionkey@0"
I20260619 20:52:26.202059 14043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:52:26.202783 14058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:26.202988 14043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:52:26.203099 14043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "30dc00e2e47a430b9d794d1897f3d985"
format_stamp: "Formatted at 2026-06-19 20:52:26 on dist-test-slave-0smx"
server_key: "184c5f0479b72463856fea536132ec68"
server_key_iv: "b5ce3b0a7410a901964fc8cb4845215c"
server_key_version: "encryptionkey@0"
I20260619 20:52:26.203190 14043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:26.217420 14043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:26.219503 14043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:26.219679 14043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:26.223982 14043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38699
I20260619 20:52:26.224011 14110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38699 every 8 connection(s)
I20260619 20:52:26.224387 14043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:26.225190 14111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:26.227398 14111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985: Bootstrap starting.
I20260619 20:52:26.227958 14111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:26.228241 14111 log.cc:824] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:26.228960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14043
I20260619 20:52:26.228992 14111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985: No bootstrap required, opened a new log
I20260619 20:52:26.229059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:52:26.229245 392 external_mini_cluster.cc:1428] Setting key 3266752e539d0e49af45c0794b18c642
I20260619 20:52:26.230906 14111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc00e2e47a430b9d794d1897f3d985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38699 } }
I20260619 20:52:26.231067 14111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:26.231112 14111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dc00e2e47a430b9d794d1897f3d985, State: Initialized, Role: FOLLOWER
I20260619 20:52:26.231235 14111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc00e2e47a430b9d794d1897f3d985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38699 } }
I20260619 20:52:26.231329 14111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:26.231374 14111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:26.231426 14111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:26.232259 14111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc00e2e47a430b9d794d1897f3d985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38699 } }
I20260619 20:52:26.232374 14111 leader_election.cc:302] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30dc00e2e47a430b9d794d1897f3d985; no voters:
I20260619 20:52:26.232571 14111 leader_election.cc:288] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:52:26.232594 14116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:26.232715 14116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [term 1 LEADER]: Becoming Leader. State: Replica: 30dc00e2e47a430b9d794d1897f3d985, State: Running, Role: LEADER
I20260619 20:52:26.232795 14116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc00e2e47a430b9d794d1897f3d985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38699 } }
I20260619 20:52:26.232846 14111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:26.233107 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30dc00e2e47a430b9d794d1897f3d985. Latest consensus state: current_term: 1 leader_uuid: "30dc00e2e47a430b9d794d1897f3d985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc00e2e47a430b9d794d1897f3d985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38699 } } }
I20260619 20:52:26.233183 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:26.233119 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30dc00e2e47a430b9d794d1897f3d985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc00e2e47a430b9d794d1897f3d985" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38699 } } }
I20260619 20:52:26.233371 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:26.233577 14121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:26.234115 14121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:26.236063 14121 catalog_manager.cc:1347] Generated new cluster ID: b7126cef31ed4539a94e0625676ac5b9
I20260619 20:52:26.236121 14121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:52:26 dist-test-slave-0smx krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:26.241247 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:26.229785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.0.98.62:38699)
0619 20:52:26.230008 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:26.230015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:26.230042 (+ 27us) server_negotiation.cc:407] Connection header received
0619 20:52:26.230909 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:26.230917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:26.230968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:26.231132 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:26.231855 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:26.232977 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:26.234117 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:26.234703 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:26.234852 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:26.237547 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:26.237551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:26.237551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:26.237562 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:26.239400 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:26.239745 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:26.239747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:26.239748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:26.239874 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:26.240048 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:26.240048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:26.240049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:26.240313 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:26.240446 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:26.240920 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 20:52:26.241046 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":117,"threads_started":1}
I20260619 20:52:26.241919 14121 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:52:26.242097 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:52:26.242585 14121 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:52:26.246085 14121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30dc00e2e47a430b9d794d1897f3d985: Generated new TSK 0
I20260619 20:52:26.246243 14121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:52:26.296119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14043
2026-06-19T20:52:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1341 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:52:26 dist-test-slave-0smx krb5kdc[14138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:52:26 dist-test-slave-0smx krb5kdc[14138](info): set up 2 sockets
Jun 19 20:52:26 dist-test-slave-0smx krb5kdc[14138](info): commencing operation
krb5kdc: starting...
W20260619 20:52:27.329525 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:52:27.440344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39435
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:27.519102 14151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:27.519277 14151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:27.519297 14151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:27.520393 14151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:27.520431 14151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:27.520445 14151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:27.520457 14151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:27.520491 14151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:27.521987 14151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:27.522249 14151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:27.522478 14151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:27.527289 14159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:27.527243 14156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:27.527307 14151 server_base.cc:1034] running on GCE node
W20260619 20:52:27.527256 14157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:27.527657 14151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:27.527916 14151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:27.529052 14151 hybrid_clock.cc:648] HybridClock initialized: now 1781902347529027 us; error 37 us; skew 500 ppm
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:27.531392 14151 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:27.531742 14151 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260619 20:52:27.531925 14151 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:27.531963 14151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:27.532058 14151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:27.533005 14151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "7caed2c800e749c483831f48f96a7c54"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "b8c09e6f23357a6cd1f6525d3c736aab"
server_key_iv: "072e97560c0bd461e0df000f03502e3e"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.533286 14151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "7caed2c800e749c483831f48f96a7c54"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "b8c09e6f23357a6cd1f6525d3c736aab"
server_key_iv: "072e97560c0bd461e0df000f03502e3e"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.534534 14151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:27.535123 14166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.535307 14151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.535434 14151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7caed2c800e749c483831f48f96a7c54"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "b8c09e6f23357a6cd1f6525d3c736aab"
server_key_iv: "072e97560c0bd461e0df000f03502e3e"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.535523 14151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:27.548771 14151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:27.550707 14151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:27.550861 14151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:27.554683 14151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39435
I20260619 20:52:27.554728 14218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39435 every 8 connection(s)
I20260619 20:52:27.555084 14151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:27.556049 14219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:27.558326 14219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap starting.
I20260619 20:52:27.558898 14219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:27.559185 14219 log.cc:824] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:27.559814 14219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: No bootstrap required, opened a new log
I20260619 20:52:27.562001 14219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:27.562134 14219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:27.562155 14219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caed2c800e749c483831f48f96a7c54, State: Initialized, Role: FOLLOWER
I20260619 20:52:27.562319 14219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:27.562398 14219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:27.562451 14219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:27.562503 14219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:27.563283 14219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:27.563390 14219 leader_election.cc:302] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caed2c800e749c483831f48f96a7c54; no voters:
I20260619 20:52:27.563548 14219 leader_election.cc:288] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:52:27.563602 14222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:27.563737 14222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 LEADER]: Becoming Leader. State: Replica: 7caed2c800e749c483831f48f96a7c54, State: Running, Role: LEADER
I20260619 20:52:27.563787 14219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:27.563813 14222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:27.564209 14224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caed2c800e749c483831f48f96a7c54. Latest consensus state: current_term: 1 leader_uuid: "7caed2c800e749c483831f48f96a7c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } } }
I20260619 20:52:27.564316 14224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:27.564193 14223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7caed2c800e749c483831f48f96a7c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } } }
I20260619 20:52:27.564893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14151
I20260619 20:52:27.564991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:52:27.565212 392 external_mini_cluster.cc:1428] Setting key 92eab445091f5046fbdc787716594081
I20260619 20:52:27.564513 14223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:27.564556 14231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:27.565972 14231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:27.567462 14231 catalog_manager.cc:1347] Generated new cluster ID: fa454e53c2c2487f981dcf27241106a8
I20260619 20:52:27.567517 14231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:27.576982 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:27.565755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.0.98.62:39435)
0619 20:52:27.566068 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:27.566077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:27.566106 (+ 29us) server_negotiation.cc:407] Connection header received
0619 20:52:27.566207 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:27.566212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:27.566252 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:27.566401 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:27.567170 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.568552 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.569629 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.570238 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.570387 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:27.573234 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:27.573238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:27.573238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:27.573250 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:27.575306 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:27.575734 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:27.575737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:27.575738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:27.575846 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:27.576050 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:27.576050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:27.576051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:27.576331 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:27.576467 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:27.576606 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 20:52:27.576727 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
I20260619 20:52:27.584095 14231 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:52:27.584592 14231 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:52:27.591076 14231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Generated new TSK 0
I20260619 20:52:27.591274 14231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:52:27.625270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39435
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:27.710440 14245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:27.710675 14245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:27.710716 14245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:27.711886 14245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:27.711951 14245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:27.711997 14245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:52:27.713526 14245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:27.713774 14245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:27.714006 14245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:27.718884 14251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:27.718930 14253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:27.719069 14245 server_base.cc:1034] running on GCE node
W20260619 20:52:27.718895 14250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:27.719326 14245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:27.719606 14245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:27.720783 14245 hybrid_clock.cc:648] HybridClock initialized: now 1781902347720771 us; error 30 us; skew 500 ppm
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:27.723842 14245 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:52:27.724370 14245 webserver.cc:460] Webserver started at http://127.0.98.1:34385/ using document root <none> and password file <none>
I20260619 20:52:27.724606 14245 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:27.724685 14245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:27.724831 14245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:27.726336 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5a2dc262cef400490d7a853b0e4d062"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "6d5eb4a63a64322f38be4541dd5a2760"
server_key_iv: "901a7e5b92c380c46bd8be8c3c456a6b"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.726747 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5a2dc262cef400490d7a853b0e4d062"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "6d5eb4a63a64322f38be4541dd5a2760"
server_key_iv: "901a7e5b92c380c46bd8be8c3c456a6b"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.728121 14245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:52:27.729041 14260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.729249 14245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:27.729411 14245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "c5a2dc262cef400490d7a853b0e4d062"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "6d5eb4a63a64322f38be4541dd5a2760"
server_key_iv: "901a7e5b92c380c46bd8be8c3c456a6b"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.729494 14245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:27.742976 14245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:27.745012 14245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:27.745208 14245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:27.745554 14245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:27.745949 14245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:27.745997 14245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.746279 14245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:27.746323 14245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.756712 14245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46087
I20260619 20:52:27.756768 14373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46087 every 8 connection(s)
I20260619 20:52:27.757467 14245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:52:27.761107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14245
I20260619 20:52:27.761240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:52:27.761605 392 external_mini_cluster.cc:1428] Setting key 47749e8c104e180512946f6bf7700d4a
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:27.770781 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:27.758750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59103 (local address 127.0.98.62:39435)
0619 20:52:27.760448 (+ 1698us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:27.760450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:27.760459 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:52:27.760487 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:27.760488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:27.760502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:27.760597 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:27.762263 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.762773 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.764203 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.764509 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.764613 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:27.767310 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:27.767311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:27.767312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:27.767321 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:27.768926 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:27.769406 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:27.769408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:27.769408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:27.769487 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:27.769831 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:27.769833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:27.769833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:27.770031 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:27.770117 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:27.770538 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 20:52:27.770666 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1662}
I20260619 20:52:27.771489 14376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:27.758629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:59103)
0619 20:52:27.758790 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:27.758830 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:27.759289 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:27.760691 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:27.760697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:27.761841 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:27.762147 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:27.762153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.762889 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.762890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:27.764048 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:27.764052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.764623 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.764624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:27.764780 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:27.765440 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:27.765451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:27.767203 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:27.769029 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:27.769036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:27.769039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:27.769270 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:27.769556 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:27.769557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:27.769557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:27.769702 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:27.770154 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:27.770161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:27.770351 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 20:52:27.771029 (+ 678us) client_negotiation.cc:240] Negotiation successful
0619 20:52:27.771233 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260619 20:52:27.772253 14374 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39435
I20260619 20:52:27.772357 14374 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:27.772557 14374 heartbeater.cc:511] Master 127.0.98.62:39435 requested a full tablet report, sending...
I20260619 20:52:27.773198 14183 ts_manager.cc:194] Registered new tserver with Master: c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:52:27.774207 14183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:52:27.791973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39435
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:27.874537 14379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:27.874764 14379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:27.874790 14379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:27.875973 14379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:27.876044 14379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:27.876102 14379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:52:27.877658 14379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:27.877909 14379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:27.878152 14379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:27.883616 14385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:27.883616 14384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:27.883631 14379 server_base.cc:1034] running on GCE node
W20260619 20:52:27.883636 14387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:27.883998 14379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:27.884274 14379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:27.885429 14379 hybrid_clock.cc:648] HybridClock initialized: now 1781902347885422 us; error 41 us; skew 500 ppm
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:27.887898 14379 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:52:27.888288 14379 webserver.cc:460] Webserver started at http://127.0.98.2:40969/ using document root <none> and password file <none>
I20260619 20:52:27.888465 14379 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:27.888504 14379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:27.888588 14379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:27.889657 14379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "4352ac050cce47b99f12220618a301b8"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "2f1db14b894edc3aa08abf1d5dcf80a4"
server_key_iv: "c9fda8cc8ca1ca12ba7e25ae28cf7b25"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.889963 14379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4352ac050cce47b99f12220618a301b8"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "2f1db14b894edc3aa08abf1d5dcf80a4"
server_key_iv: "c9fda8cc8ca1ca12ba7e25ae28cf7b25"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.891225 14379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:52:27.891907 14394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.892130 14379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:27.892220 14379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "4352ac050cce47b99f12220618a301b8"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "2f1db14b894edc3aa08abf1d5dcf80a4"
server_key_iv: "c9fda8cc8ca1ca12ba7e25ae28cf7b25"
server_key_version: "encryptionkey@0"
I20260619 20:52:27.892270 14379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:27.903339 14379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:27.905764 14379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:27.905884 14379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:27.906073 14379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:27.906364 14379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:27.906396 14379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.906418 14379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:27.906431 14379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:27.912025 14379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35003
I20260619 20:52:27.912111 14507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35003 every 8 connection(s)
I20260619 20:52:27.912431 14379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:52:27.916908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14379
I20260619 20:52:27.917014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:52:27.917260 392 external_mini_cluster.cc:1428] Setting key 05379b61a364f6108aa0953777e5aa8e
Jun 19 20:52:27 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:27.922747 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:27.913225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52531 (local address 127.0.98.62:39435)
0619 20:52:27.913285 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:27.913327 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:27.913537 (+ 210us) server_negotiation.cc:407] Connection header received
0619 20:52:27.914111 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:27.914112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:27.914127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:27.914192 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:27.915213 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.915745 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.916880 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.917177 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.917238 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:27.919790 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:27.919791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:27.919792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:27.919799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:27.921366 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:27.921786 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:27.921786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:27.921787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:27.921849 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:27.922108 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:27.922109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:27.922109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:27.922223 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:27.922305 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:27.922568 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 20:52:27.922655 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:27.922955 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:27.913196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:52531)
0619 20:52:27.913398 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:27.913444 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:27.913992 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:27.914269 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:27.914273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:27.914874 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:27.915072 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:27.915076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.915863 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.915865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:27.916778 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:27.916781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:27.917270 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:27.917271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:27.917469 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:27.918012 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:27.918023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:27.919691 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:27.921468 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:27.921475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:27.921477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:27.921700 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:27.921917 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:27.921918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:27.921918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:27.922031 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:27.922290 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:27.922296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:27.922440 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 20:52:27.922548 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 20:52:27.922704 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260619 20:52:27.923483 14508 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39435
I20260619 20:52:27.923590 14508 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:27.923775 14508 heartbeater.cc:511] Master 127.0.98.62:39435 requested a full tablet report, sending...
I20260619 20:52:27.924242 14183 ts_manager.cc:194] Registered new tserver with Master: 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003)
I20260619 20:52:27.924782 14183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52531
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:52:27.943845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39435
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:28.025197 14513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:28.025424 14513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:28.025446 14513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:28.026507 14513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:28.026540 14513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:28.026566 14513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:52:28.027933 14513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:28.028162 14513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:28.028406 14513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:28.033375 14521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.033406 14518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.033497 14513 server_base.cc:1034] running on GCE node
W20260619 20:52:28.033375 14519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.033962 14513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:28.034235 14513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:28.035385 14513 hybrid_clock.cc:648] HybridClock initialized: now 1781902348035361 us; error 35 us; skew 500 ppm
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:28.038398 14513 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:52:28.038898 14513 webserver.cc:460] Webserver started at http://127.0.98.3:37375/ using document root <none> and password file <none>
I20260619 20:52:28.039148 14513 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:28.039215 14513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:28.039358 14513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:28.040390 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbc4e6142e0e4dc2a75774032bad566c"
format_stamp: "Formatted at 2026-06-19 20:52:28 on dist-test-slave-0smx"
server_key: "fb0f9b681c9cd6fb44dcb48b22646446"
server_key_iv: "53845c06ea864c4cc793d076e81ab261"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.040750 14513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbc4e6142e0e4dc2a75774032bad566c"
format_stamp: "Formatted at 2026-06-19 20:52:28 on dist-test-slave-0smx"
server_key: "fb0f9b681c9cd6fb44dcb48b22646446"
server_key_iv: "53845c06ea864c4cc793d076e81ab261"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.042117 14513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:52:28.042840 14528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.043001 14513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:28.043097 14513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "dbc4e6142e0e4dc2a75774032bad566c"
format_stamp: "Formatted at 2026-06-19 20:52:28 on dist-test-slave-0smx"
server_key: "fb0f9b681c9cd6fb44dcb48b22646446"
server_key_iv: "53845c06ea864c4cc793d076e81ab261"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.043184 14513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:28.070998 14513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:28.072824 14513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:28.073002 14513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:28.073246 14513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:28.073570 14513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:28.073624 14513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.073669 14513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:28.073693 14513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.079353 14513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43971
I20260619 20:52:28.079429 14641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43971 every 8 connection(s)
I20260619 20:52:28.079762 14513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:28.088452 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.080486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47501 (local address 127.0.98.62:39435)
0619 20:52:28.080556 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.080557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.080739 (+ 182us) server_negotiation.cc:407] Connection header received
0619 20:52:28.081210 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.081211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.081220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.081283 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.082230 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.082716 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.083655 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.083832 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.083882 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.085879 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.085881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.085881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.085887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.087319 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.087672 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.087673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.087673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.087721 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.087931 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.087931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.087931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.088038 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.088088 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.088324 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.088385 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:52:28.088686 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.080468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:47501)
0619 20:52:28.080612 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.080652 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.081130 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.081386 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.081389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.081960 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:28.082148 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.082153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.082800 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.082800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.083583 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.083584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.083902 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.083902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.083997 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.084427 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:28.084435 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:28.085796 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:28.087394 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.087400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.087402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.087608 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.087787 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.087787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.087787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.087878 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.088092 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:28.088095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:28.088206 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.088291 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.088407 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260619 20:52:28.088956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14513
I20260619 20:52:28.089046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:52:28.089226 392 external_mini_cluster.cc:1428] Setting key d125b14236b6fcd16ef69ea1084e4e6c
I20260619 20:52:28.089243 14642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39435
I20260619 20:52:28.089368 14642 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:28.089588 14642 heartbeater.cc:511] Master 127.0.98.62:39435 requested a full tablet report, sending...
I20260619 20:52:28.089990 14182 ts_manager.cc:194] Registered new tserver with Master: dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
I20260619 20:52:28.090371 14182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47501
I20260619 20:52:28.101055 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:52:28.108305 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.102428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.0.98.62:39435)
0619 20:52:28.102491 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.102493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.102700 (+ 207us) server_negotiation.cc:407] Connection header received
0619 20:52:28.102767 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.102767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.102781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.102827 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.103398 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.103871 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.104634 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.104819 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.104884 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.106123 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.106125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.106125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.106132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.107352 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.107648 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.107649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.107650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.107703 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.107861 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.107861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.107862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.107993 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.108066 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.108154 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.108210 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:52:28.110047 14182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:52:28.111591 14182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:52:28.117995 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.113968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43971 (local address 127.0.0.1:36522)
0619 20:52:28.114345 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.114353 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.114415 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.114718 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.114721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.114723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.114785 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.114787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.115647 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.115648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.116819 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.116821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.117708 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.117709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.117787 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.117800 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.117873 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.117924 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260619 20:52:28.118006 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.114141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.98.3:43971)
0619 20:52:28.114313 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.114319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.114498 (+ 179us) server_negotiation.cc:407] Connection header received
0619 20:52:28.114539 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.114541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.114558 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.114628 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.114861 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.115551 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.116900 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.117626 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.117718 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.117780 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.117887 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.117923 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260619 20:52:28.118435 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.113968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35003 (local address 127.0.0.1:59040)
0619 20:52:28.114305 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.114347 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.114415 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.114732 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.114733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.114737 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.114801 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.114804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.116261 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.116262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.117375 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.117377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.118212 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.118212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.118271 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.118275 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.118315 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.118364 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":14,"thread_start_us":59,"threads_started":1}
I20260619 20:52:28.118459 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.114074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.98.2:35003)
0619 20:52:28.114477 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.114482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.114491 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:52:28.114535 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.114537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.114552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.114630 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.114884 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.116163 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.117463 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.118136 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.118193 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.118243 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.118334 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.118380 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":86,"threads_started":1}
I20260619 20:52:28.118590 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.114169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46087 (local address 127.0.0.1:33150)
0619 20:52:28.114372 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.114383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.114439 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.114732 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.114733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.114735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.114801 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.114804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.116133 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.116134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.117104 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.117106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.118065 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.118066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.118429 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.118433 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.118465 (+ 32us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.118507 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":6,"thread_start_us":68,"threads_started":1}
I20260619 20:52:28.118901 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.114283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.0.98.1:46087)
0619 20:52:28.114552 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.114556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.114564 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:52:28.114601 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.114602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.114617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.114881 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.114953 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.116022 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.117200 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.117962 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.118690 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.118756 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.118791 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.118830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":74,"threads_started":1}
I20260619 20:52:28.119133 14576 tablet_service.cc:1467] Processing CreateTablet for tablet 42d72b550df24374950c02aeb12fa645 (DEFAULT_TABLE table=test-table [id=6733e003ff1b4e76819853bbd7e0ef59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:28.119482 14576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d72b550df24374950c02aeb12fa645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:28.119475 14305 tablet_service.cc:1467] Processing CreateTablet for tablet 42d72b550df24374950c02aeb12fa645 (DEFAULT_TABLE table=test-table [id=6733e003ff1b4e76819853bbd7e0ef59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:28.119810 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d72b550df24374950c02aeb12fa645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:28.121734 14442 tablet_service.cc:1467] Processing CreateTablet for tablet 42d72b550df24374950c02aeb12fa645 (DEFAULT_TABLE table=test-table [id=6733e003ff1b4e76819853bbd7e0ef59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:28.122007 14659 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Bootstrap starting.
I20260619 20:52:28.122058 14442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d72b550df24374950c02aeb12fa645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:28.122486 14660 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap starting.
I20260619 20:52:28.122866 14659 tablet_bootstrap.cc:654] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:28.123095 14660 tablet_bootstrap.cc:654] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:28.123507 14659 log.cc:824] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.123667 14660 log.cc:824] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.124289 14659 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: No bootstrap required, opened a new log
I20260619 20:52:28.124306 14660 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: No bootstrap required, opened a new log
I20260619 20:52:28.124361 14660 ts_tablet_manager.cc:1397] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:28.124361 14659 ts_tablet_manager.cc:1397] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:52:28.124933 14663 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Bootstrap starting.
I20260619 20:52:28.125595 14663 tablet_bootstrap.cc:654] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:28.126179 14663 log.cc:824] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.126291 14659 raft_consensus.cc:348] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.126296 14660 raft_consensus.cc:348] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.126468 14659 raft_consensus.cc:374] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:28.126468 14660 raft_consensus.cc:374] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:28.126510 14659 raft_consensus.cc:729] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc4e6142e0e4dc2a75774032bad566c, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.126510 14660 raft_consensus.cc:729] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a2dc262cef400490d7a853b0e4d062, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.126616 14660 consensus_queue.cc:260] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.126631 14659 consensus_queue.cc:260] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.126899 14659 ts_tablet_manager.cc:1428] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:52:28.126899 14660 ts_tablet_manager.cc:1428] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:52:28.126914 14663 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: No bootstrap required, opened a new log
I20260619 20:52:28.127022 14642 heartbeater.cc:503] Master 127.0.98.62:39435 was elected leader, sending a full tablet report...
I20260619 20:52:28.127046 14374 heartbeater.cc:503] Master 127.0.98.62:39435 was elected leader, sending a full tablet report...
I20260619 20:52:28.127264 14663 ts_tablet_manager.cc:1397] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:52:28.128981 14663 raft_consensus.cc:348] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.129132 14663 raft_consensus.cc:374] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:28.129179 14663 raft_consensus.cc:729] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4352ac050cce47b99f12220618a301b8, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.129321 14663 consensus_queue.cc:260] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.129586 14663 ts_tablet_manager.cc:1428] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:52:28.129643 14508 heartbeater.cc:503] Master 127.0.98.62:39435 was elected leader, sending a full tablet report...
I20260619 20:52:28.145435 14665 raft_consensus.cc:482] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:28.145521 14665 raft_consensus.cc:504] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.145874 14665 leader_election.cc:288] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087)
I20260619 20:52:28.148634 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.145994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35003 (local address 127.0.98.3:60035)
0619 20:52:28.146055 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.146068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.146125 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.146287 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.146288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.146290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.146358 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.146359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.147030 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.147031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.147820 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.147822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.148397 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.148398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.148477 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.148482 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.148535 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.148581 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:52:28.148653 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.146072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60035 (local address 127.0.98.2:35003)
0619 20:52:28.146111 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.146113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.146123 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:52:28.146180 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.146181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.146192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.146257 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.146439 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.146958 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.147911 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.148319 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.148371 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.148379 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.148556 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.148590 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:28.148838 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.146097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54175 (local address 127.0.98.1:46087)
0619 20:52:28.146132 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.146134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.146282 (+ 148us) server_negotiation.cc:407] Connection header received
0619 20:52:28.146353 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.146354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.146365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.146418 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.146571 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.147129 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.148049 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.148460 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.148532 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.148542 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.148741 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.148786 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:52:28.149026 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.146053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46087 (local address 127.0.98.3:54175)
0619 20:52:28.146208 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.146218 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.146286 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.146427 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.146428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.146430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.146506 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.146508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.147203 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.147204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.147967 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.147969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.148561 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.148562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.148641 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.148645 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.148933 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.148976 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260619 20:52:28.149005 14462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d72b550df24374950c02aeb12fa645" candidate_uuid: "dbc4e6142e0e4dc2a75774032bad566c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4352ac050cce47b99f12220618a301b8" is_pre_election: true
I20260619 20:52:28.149101 14462 raft_consensus.cc:2454] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc4e6142e0e4dc2a75774032bad566c in term 0.
I20260619 20:52:28.149326 14529 leader_election.cc:302] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c; no voters:
I20260619 20:52:28.149370 14328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d72b550df24374950c02aeb12fa645" candidate_uuid: "dbc4e6142e0e4dc2a75774032bad566c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a2dc262cef400490d7a853b0e4d062" is_pre_election: true
I20260619 20:52:28.149463 14665 raft_consensus.cc:2781] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:52:28.149493 14328 raft_consensus.cc:2454] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc4e6142e0e4dc2a75774032bad566c in term 0.
I20260619 20:52:28.149507 14665 raft_consensus.cc:482] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:52:28.149531 14665 raft_consensus.cc:3037] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:28.150238 14665 raft_consensus.cc:504] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.150370 14665 leader_election.cc:288] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [CANDIDATE]: Term 1 election: Requested vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087)
I20260619 20:52:28.150521 14462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d72b550df24374950c02aeb12fa645" candidate_uuid: "dbc4e6142e0e4dc2a75774032bad566c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4352ac050cce47b99f12220618a301b8"
I20260619 20:52:28.150549 14328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d72b550df24374950c02aeb12fa645" candidate_uuid: "dbc4e6142e0e4dc2a75774032bad566c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a2dc262cef400490d7a853b0e4d062"
I20260619 20:52:28.150602 14328 raft_consensus.cc:3037] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:28.150615 14462 raft_consensus.cc:3037] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:28.151257 14328 raft_consensus.cc:2454] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc4e6142e0e4dc2a75774032bad566c in term 1.
I20260619 20:52:28.151366 14462 raft_consensus.cc:2454] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc4e6142e0e4dc2a75774032bad566c in term 1.
I20260619 20:52:28.151463 14531 leader_election.cc:302] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5a2dc262cef400490d7a853b0e4d062, dbc4e6142e0e4dc2a75774032bad566c; no voters:
I20260619 20:52:28.151547 14665 raft_consensus.cc:2781] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:28.151686 14665 raft_consensus.cc:686] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 1 LEADER]: Becoming Leader. State: Replica: dbc4e6142e0e4dc2a75774032bad566c, State: Running, Role: LEADER
I20260619 20:52:28.151793 14665 consensus_queue.cc:237] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.152485 14182 catalog_manager.cc:5495] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c reported cstate change: term changed from 0 to 1, leader changed from <none> to dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dbc4e6142e0e4dc2a75774032bad566c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } health_report { overall_health: UNKNOWN } } }
W20260619 20:52:28.162976 14509 tablet.cc:2369] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:52:28.163160 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.161015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.3:43971)
0619 20:52:28.161054 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.161056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.161063 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:52:28.161140 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.161141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.161156 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.161211 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:52:28.161367 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.161802 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.162486 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.162669 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.162717 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.162838 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:52:28.162921 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:52:28.162973 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.163063 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.163082 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:52:28.164748 14462 raft_consensus.cc:1261] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Refusing update from remote peer dbc4e6142e0e4dc2a75774032bad566c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:28.164810 14328 raft_consensus.cc:1261] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Refusing update from remote peer dbc4e6142e0e4dc2a75774032bad566c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:28.165048 14665 consensus_queue.cc:1035] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:28.165202 14669 consensus_queue.cc:1035] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:28.167330 14672 mvcc.cc:205] Tried to move back new op lower bound from 7298672018080780288 to 7298672018030354432. Current Snapshot: MvccSnapshot[applied={T|T < 7298672018080780288}]
I20260619 20:52:28.167482 14673 mvcc.cc:205] Tried to move back new op lower bound from 7298672018080780288 to 7298672018030354432. Current Snapshot: MvccSnapshot[applied={T|T < 7298672018080780288}]
I20260619 20:52:28.167709 14674 mvcc.cc:205] Tried to move back new op lower bound from 7298672018080780288 to 7298672018030354432. Current Snapshot: MvccSnapshot[applied={T|T < 7298672018080780288}]
I20260619 20:52:28.169819 14181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:28.169904 14181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:28.170580 14181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Sending DeleteTablet for 3 replicas of tablet 42d72b550df24374950c02aeb12fa645
I20260619 20:52:28.170897 14576 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d72b550df24374950c02aeb12fa645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36522
I20260619 20:52:28.170935 14305 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d72b550df24374950c02aeb12fa645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33150
I20260619 20:52:28.171110 14682 tablet_replica.cc:330] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: stopping tablet replica
I20260619 20:52:28.171111 14442 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d72b550df24374950c02aeb12fa645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59040
I20260619 20:52:28.171207 14682 raft_consensus.cc:2229] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:52:28.171330 14682 raft_consensus.cc:2258] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:28.171334 14684 tablet_replica.cc:330] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: stopping tablet replica
I20260619 20:52:28.171366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14245
I20260619 20:52:28.171775 14682 ts_tablet_manager.cc:1905] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:28.172173 14684 raft_consensus.cc:2229] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:28.172329 14684 raft_consensus.cc:2258] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:28.172818 14684 ts_tablet_manager.cc:1905] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:28.173691 14682 ts_tablet_manager.cc:1918] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:28.173774 14682 log.cc:1196] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/42d72b550df24374950c02aeb12fa645
I20260619 20:52:28.174049 14682 ts_tablet_manager.cc:1939] T 42d72b550df24374950c02aeb12fa645 P dbc4e6142e0e4dc2a75774032bad566c: Deleting consensus metadata
I20260619 20:52:28.174407 14169 catalog_manager.cc:4841] TS dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): tablet 42d72b550df24374950c02aeb12fa645 (table test-table [id=6733e003ff1b4e76819853bbd7e0ef59]) successfully deleted
I20260619 20:52:28.174754 14684 ts_tablet_manager.cc:1918] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:28.174876 14684 log.cc:1196] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/42d72b550df24374950c02aeb12fa645
I20260619 20:52:28.175211 14684 ts_tablet_manager.cc:1939] T 42d72b550df24374950c02aeb12fa645 P 4352ac050cce47b99f12220618a301b8: Deleting consensus metadata
I20260619 20:52:28.175609 14167 catalog_manager.cc:4841] TS 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003): tablet 42d72b550df24374950c02aeb12fa645 (table test-table [id=6733e003ff1b4e76819853bbd7e0ef59]) successfully deleted
W20260619 20:52:28.177809 14169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46087 (error 108)
I20260619 20:52:28.178334 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.178200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46087 (local address 127.0.0.1:33164)
0619 20:52:28.178259 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.178303 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:52:28.178401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14379
W20260619 20:52:28.179174 14169 catalog_manager.cc:4584] TS c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42d72b550df24374950c02aeb12fa645: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46087: connect: Connection refused (error 111)
I20260619 20:52:28.183301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14513
I20260619 20:52:28.187345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14151
I20260619 20:52:28.192409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39435
--webserver_interface=127.0.98.62
--webserver_port=42597
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:28.272292 14686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:28.272491 14686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:28.272514 14686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:28.273625 14686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:28.273661 14686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:28.273675 14686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:28.273687 14686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:28.273700 14686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:28.275063 14686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:28.275245 14686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:28.275406 14686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:28.283895 14695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.284076 14686 server_base.cc:1034] running on GCE node
W20260619 20:52:28.283895 14692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.283970 14693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.284478 14686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:28.284843 14686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:28.285985 14686 hybrid_clock.cc:648] HybridClock initialized: now 1781902348285974 us; error 27 us; skew 500 ppm
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:28.288486 14686 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:28.288941 14686 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260619 20:52:28.289184 14686 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:28.289269 14686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:28.290733 14686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:28.291450 14702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.291635 14686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:28.291741 14686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7caed2c800e749c483831f48f96a7c54"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "b8c09e6f23357a6cd1f6525d3c736aab"
server_key_iv: "072e97560c0bd461e0df000f03502e3e"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.292146 14686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:28.302910 14686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:28.304702 14686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:28.304869 14686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:28.308749 14686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39435
I20260619 20:52:28.308799 14754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39435 every 8 connection(s)
I20260619 20:52:28.309198 14686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:28.310828 14755 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:52:28.311471 14755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap starting.
I20260619 20:52:28.316852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14686
I20260619 20:52:28.317193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46087
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39435
--webserver_port=34385
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:28.320175 14755 log.cc:824] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.322229 14755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:52:28.322522 14755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap complete.
I20260619 20:52:28.324856 14755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:28.325079 14755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caed2c800e749c483831f48f96a7c54, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.325268 14755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:28.325384 14755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:28.325440 14755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:28.325491 14755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:52:28.326505 14755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:28.326591 14755 leader_election.cc:302] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caed2c800e749c483831f48f96a7c54; no voters:
I20260619 20:52:28.326815 14755 leader_election.cc:288] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:52:28.326849 14759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:52:28.326960 14759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 LEADER]: Becoming Leader. State: Replica: 7caed2c800e749c483831f48f96a7c54, State: Running, Role: LEADER
I20260619 20:52:28.327078 14755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:28.327044 14759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:28.327411 14760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7caed2c800e749c483831f48f96a7c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } } }
I20260619 20:52:28.327490 14760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:28.327694 14761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caed2c800e749c483831f48f96a7c54. Latest consensus state: current_term: 2 leader_uuid: "7caed2c800e749c483831f48f96a7c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } } }
I20260619 20:52:28.327804 14761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:28.327939 14765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:28.328750 14765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:28.328867 14765 catalog_manager.cc:1259] Loaded cluster ID: fa454e53c2c2487f981dcf27241106a8
I20260619 20:52:28.329087 14765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:52:28.329367 14765 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 20:52:28.332280 14765 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 20:52:28.338351 14765 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:52:28.338570 14765 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Loaded TSK: 0
I20260619 20:52:28.338809 14765 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:52:28.408035 14757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:28.408282 14757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:28.408334 14757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:28.409526 14757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:28.409587 14757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:28.409636 14757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:52:28.411079 14757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34385
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:28.411331 14757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:28.411561 14757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:28.417455 14783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.417655 14757 server_base.cc:1034] running on GCE node
W20260619 20:52:28.417512 14785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.417734 14782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.417948 14757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:28.418210 14757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:28.419368 14757 hybrid_clock.cc:648] HybridClock initialized: now 1781902348419360 us; error 25 us; skew 500 ppm
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:28.427204 14757 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:52:28.427752 14757 webserver.cc:460] Webserver started at http://127.0.98.1:34385/ using document root <none> and password file <none>
I20260619 20:52:28.428035 14757 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:28.428109 14757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:28.429903 14757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:52:28.430701 14792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.430881 14757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:28.430965 14757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "c5a2dc262cef400490d7a853b0e4d062"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "6d5eb4a63a64322f38be4541dd5a2760"
server_key_iv: "901a7e5b92c380c46bd8be8c3c456a6b"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.431324 14757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:28.439260 14757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:28.441399 14757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:28.441546 14757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:28.441772 14757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:28.442201 14799 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:52:28.444078 14757 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:52:28.444136 14757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:52:28.444180 14757 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:52:28.444794 14757 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:52:28.444846 14757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:52:28.444912 14799 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap starting.
I20260619 20:52:28.451850 14757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46087
I20260619 20:52:28.451983 14906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46087 every 8 connection(s)
I20260619 20:52:28.452263 14757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:52:28.452593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14757
I20260619 20:52:28.452881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35003
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39435
--webserver_port=40969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:28.455489 14799 log.cc:824] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.458602 14799 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:52:28.458902 14799 tablet_bootstrap.cc:492] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap complete.
I20260619 20:52:28.460088 14799 ts_tablet_manager.cc:1397] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260619 20:52:28.461426 14799 raft_consensus.cc:348] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.461737 14799 raft_consensus.cc:729] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a2dc262cef400490d7a853b0e4d062, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.461879 14799 consensus_queue.cc:260] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.462153 14799 ts_tablet_manager.cc:1428] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:28.471316 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.456175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:33129)
0619 20:52:28.457603 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.457652 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.458924 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.459635 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.459640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.460057 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:28.460424 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.460431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.462560 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.462561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.463903 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.463906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.464703 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.464704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.464840 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.465548 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:28.465558 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:28.467387 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:28.469527 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.469533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.469535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.469822 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.470126 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.470127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.470127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.470250 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.470669 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:28.470673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:28.470820 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.470928 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.471082 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":76,"threads_started":1}
I20260619 20:52:28.471316 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.456411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33129 (local address 127.0.98.62:39435)
0619 20:52:28.456604 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.456616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.457787 (+ 1171us) server_negotiation.cc:407] Connection header received
0619 20:52:28.459230 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.459239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.459308 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.459426 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.460567 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.462437 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.464008 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.464603 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.464719 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.467493 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.467497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.467498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.467509 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.469444 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.469914 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.469917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.469918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.470039 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.470320 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.470321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.470322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.470599 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.470680 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.470995 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.471128 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260619 20:52:28.472927 14907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39435
I20260619 20:52:28.473026 14907 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:28.473207 14907 heartbeater.cc:511] Master 127.0.98.62:39435 requested a full tablet report, sending...
I20260619 20:52:28.473995 14719 ts_manager.cc:194] Registered new tserver with Master: c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087)
I20260619 20:52:28.475186 14719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33129
I20260619 20:52:28.476064 14907 heartbeater.cc:503] Master 127.0.98.62:39435 was elected leader, sending a full tablet report...
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:52:28.483160 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.474586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46087 (local address 127.0.0.1:33168)
0619 20:52:28.474950 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.474983 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.475041 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.475410 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.475482 (+ 72us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.476457 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:28.476501 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.476503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.477490 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.477491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.478053 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.478055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.478515 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.478518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.478594 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.479071 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:28.479076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:28.480270 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:28.482007 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.482013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.482015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.482211 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.482436 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.482437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.482437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.482503 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.482780 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:28.482786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:28.482837 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.482949 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.483060 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":86,"threads_started":1}
I20260619 20:52:28.483160 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.474813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.0.98.1:46087)
0619 20:52:28.475161 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.475165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.475175 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:52:28.475207 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.475208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.475222 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.475350 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.476619 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.477408 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.478153 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.478436 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.478516 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.480361 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.480368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.480369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.480375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.481904 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.482308 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.482311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.482311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.482367 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.482567 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.482568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.482568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.482705 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.482798 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.482983 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.483067 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":63,"threads_started":1}
I20260619 20:52:28.483572 14841 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d72b550df24374950c02aeb12fa645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33168
I20260619 20:52:28.484196 14916 tablet_replica.cc:330] stopping tablet replica
I20260619 20:52:28.484310 14916 raft_consensus.cc:2229] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:28.484397 14916 raft_consensus.cc:2258] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:28.484668 14916 ts_tablet_manager.cc:1905] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:28.486706 14916 ts_tablet_manager.cc:1918] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:28.486793 14916 log.cc:1196] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/42d72b550df24374950c02aeb12fa645
I20260619 20:52:28.487036 14916 ts_tablet_manager.cc:1939] T 42d72b550df24374950c02aeb12fa645 P c5a2dc262cef400490d7a853b0e4d062: Deleting consensus metadata
I20260619 20:52:28.487531 14705 catalog_manager.cc:4841] TS c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087): tablet 42d72b550df24374950c02aeb12fa645 (table test-table [id=6733e003ff1b4e76819853bbd7e0ef59]) successfully deleted
W20260619 20:52:28.540944 14909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:28.541129 14909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:28.541152 14909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:28.542343 14909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:28.542384 14909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:28.542409 14909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:52:28.543884 14909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40969
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:28.544111 14909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:28.544342 14909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:28.549126 14922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.549185 14921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.549175 14924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.549343 14909 server_base.cc:1034] running on GCE node
I20260619 20:52:28.549598 14909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:28.549865 14909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:28.551002 14909 hybrid_clock.cc:648] HybridClock initialized: now 1781902348550985 us; error 36 us; skew 500 ppm
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:28.553519 14909 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:52:28.553972 14909 webserver.cc:460] Webserver started at http://127.0.98.2:40969/ using document root <none> and password file <none>
I20260619 20:52:28.554199 14909 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:28.554255 14909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:28.555573 14909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:52:28.556138 14931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.556313 14909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:52:28.556371 14909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "4352ac050cce47b99f12220618a301b8"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "2f1db14b894edc3aa08abf1d5dcf80a4"
server_key_iv: "c9fda8cc8ca1ca12ba7e25ae28cf7b25"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.556695 14909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:28.573093 14909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:28.574923 14909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:28.575079 14909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:28.575309 14909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:28.575618 14909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:28.575671 14909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.575711 14909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:28.575747 14909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.581633 14909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35003
I20260619 20:52:28.581681 15044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35003 every 8 connection(s)
I20260619 20:52:28.582008 14909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:28.588075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 14909
I20260619 20:52:28.588326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39435
--webserver_port=37375
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:28.590975 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.582725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44981 (local address 127.0.98.62:39435)
0619 20:52:28.582804 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.582806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.582981 (+ 175us) server_negotiation.cc:407] Connection header received
0619 20:52:28.583449 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.583450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.583460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.583516 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.584380 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.584874 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.585928 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.586122 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.586175 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.588105 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.588107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.588108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.588115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.589592 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.590012 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.590013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.590014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.590072 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.590323 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.590324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.590325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.590437 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.590512 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.590844 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.590911 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 20:52:28.591248 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.582708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:44981)
0619 20:52:28.582853 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.582893 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.583369 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.583567 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.583570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.584117 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:28.584308 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.584312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.584950 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.584951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.585856 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.585859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.586193 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.586194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.586288 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.586695 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:28.586702 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:28.588006 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:28.589696 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.589703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.589706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.589929 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.590142 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.590142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.590143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.590253 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.590519 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:28.590526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:28.590686 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.590811 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.590982 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260619 20:52:28.591727 15045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39435
I20260619 20:52:28.591820 15045 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:28.592016 15045 heartbeater.cc:511] Master 127.0.98.62:39435 requested a full tablet report, sending...
I20260619 20:52:28.592393 14719 ts_manager.cc:194] Registered new tserver with Master: 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003)
I20260619 20:52:28.592769 14719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44981
W20260619 20:52:28.670132 15048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:28.670374 15048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:28.670411 15048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:28.671540 15048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:28.671605 15048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:28.671650 15048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:52:28.673121 15048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37375
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:28.673404 15048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:28.673633 15048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:28.678462 15056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.678529 15053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:28.678457 15054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:28.678666 15048 server_base.cc:1034] running on GCE node
I20260619 20:52:28.678921 15048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:28.679147 15048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:28.680311 15048 hybrid_clock.cc:648] HybridClock initialized: now 1781902348680307 us; error 37 us; skew 500 ppm
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:28.682932 15048 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:52:28.683419 15048 webserver.cc:460] Webserver started at http://127.0.98.3:37375/ using document root <none> and password file <none>
I20260619 20:52:28.683647 15048 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:28.683732 15048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:28.685025 15048 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:28.685680 15063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.685848 15048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.685945 15048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "dbc4e6142e0e4dc2a75774032bad566c"
format_stamp: "Formatted at 2026-06-19 20:52:28 on dist-test-slave-0smx"
server_key: "fb0f9b681c9cd6fb44dcb48b22646446"
server_key_iv: "53845c06ea864c4cc793d076e81ab261"
server_key_version: "encryptionkey@0"
I20260619 20:52:28.686303 15048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:28.694717 15048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:28.697064 15048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:28.697232 15048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:28.697490 15048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:28.697798 15048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:28.697858 15048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.697899 15048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:28.697926 15048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:28.703738 15048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43971
I20260619 20:52:28.703810 15176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43971 every 8 connection(s)
I20260619 20:52:28.704144 15048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:52:28 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:28.713492 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.704872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45069 (local address 127.0.98.62:39435)
0619 20:52:28.704924 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.704927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.705164 (+ 237us) server_negotiation.cc:407] Connection header received
0619 20:52:28.705699 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.705700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.705710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.705762 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.706659 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.707175 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.708258 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.708466 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.708527 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.710751 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.710752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.710753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.710759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.712175 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.712588 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.712589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.712589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.712637 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.712874 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.712874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.712874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.712973 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.713024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.713356 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.713427 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:52:28.713627 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.704878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:45069)
0619 20:52:28.705041 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.705075 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.705600 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.705814 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.705817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.706369 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:28.706558 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.706562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.707278 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.707279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.708167 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.708170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.708560 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.708561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.708681 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.709144 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:28.709152 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:28.710668 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:28.712262 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.712269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.712270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.712508 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.712694 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:28.712694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:28.712695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:28.712797 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:28.713050 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:28.713056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:28.713186 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.713285 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.713427 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260619 20:52:28.713805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15048
I20260619 20:52:28.714159 15177 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39435
I20260619 20:52:28.714260 15177 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:28.714445 15177 heartbeater.cc:511] Master 127.0.98.62:39435 requested a full tablet report, sending...
I20260619 20:52:28.714821 14719 ts_manager.cc:194] Registered new tserver with Master: dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
I20260619 20:52:28.715193 14719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45069
I20260619 20:52:28.720052 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.714179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.0.98.62:39435)
0619 20:52:28.714234 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.714236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.714354 (+ 118us) server_negotiation.cc:407] Connection header received
0619 20:52:28.714418 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.714419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.714428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.714493 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:28.715162 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.715675 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.716422 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.716616 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.716688 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.717882 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:28.717883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:28.717883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:28.717889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:28.718967 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.719306 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.719307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.719308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.719359 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:28.719552 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:28.719553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:28.719553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:28.719657 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:28.719850 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.719886 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.719949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:52:28.720762 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:52:28.723537 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.721106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.98.62:39435)
0619 20:52:28.721144 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.721147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.721271 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:52:28.721385 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.721386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.721420 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.721505 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:52:28.721621 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.722065 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.722796 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.722988 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.723071 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.723244 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:52:28.723344 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:52:28.723435 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.723457 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.723471 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:52:28.723819 14719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:52:28.725574 14719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:52:28.727949 14841 tablet_service.cc:1467] Processing CreateTablet for tablet 432e1935054742b0a3513ec1bdd1f247 (DEFAULT_TABLE table=test-table [id=e42e8820267e4509996e6dc2eb23ae92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:28.728195 14841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432e1935054742b0a3513ec1bdd1f247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:28.729696 14799 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap starting.
I20260619 20:52:28.730049 14799 tablet_bootstrap.cc:654] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:28.730640 14799 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: No bootstrap required, opened a new log
I20260619 20:52:28.730705 14799 ts_tablet_manager.cc:1397] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:28.731024 14799 raft_consensus.cc:348] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.731107 14799 raft_consensus.cc:374] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:28.731158 14799 raft_consensus.cc:729] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a2dc262cef400490d7a853b0e4d062, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.731204 14799 consensus_queue.cc:260] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.731376 14799 ts_tablet_manager.cc:1428] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:28.731639 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.727460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35003 (local address 127.0.0.1:59050)
0619 20:52:28.727665 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.727675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.727725 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.728007 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.728008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.728010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.728054 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.728056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.729134 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.729135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.730366 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.730369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.731364 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.731365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.731428 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.731446 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.731493 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.731540 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":24,"thread_start_us":55,"threads_started":1}
I20260619 20:52:28.731653 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.727622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.98.2:35003)
0619 20:52:28.727809 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.727814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.727823 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:52:28.727882 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.727884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.727900 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.727980 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.728109 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.729030 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.730464 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.731198 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.731250 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.731305 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.731512 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.731565 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
I20260619 20:52:28.732326 14979 tablet_service.cc:1467] Processing CreateTablet for tablet 432e1935054742b0a3513ec1bdd1f247 (DEFAULT_TABLE table=test-table [id=e42e8820267e4509996e6dc2eb23ae92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:28.732661 14979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432e1935054742b0a3513ec1bdd1f247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:28.733843 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.727460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43971 (local address 127.0.0.1:36536)
0619 20:52:28.727514 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.727527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.727572 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.730078 (+ 2506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.730079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.730081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.730125 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.730127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.731613 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.731614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.732703 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.732706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.733615 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.733616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.733666 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.733670 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.733714 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.733758 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:52:28.733843 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.729716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.98.3:43971)
0619 20:52:28.729906 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.729910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.729920 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:52:28.729958 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.729960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.729975 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.730063 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.730348 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.731508 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.732806 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.733537 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.733617 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.733671 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.733728 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.733764 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260619 20:52:28.734534 15111 tablet_service.cc:1467] Processing CreateTablet for tablet 432e1935054742b0a3513ec1bdd1f247 (DEFAULT_TABLE table=test-table [id=e42e8820267e4509996e6dc2eb23ae92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:28.734844 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432e1935054742b0a3513ec1bdd1f247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:28.735888 15186 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Bootstrap starting.
I20260619 20:52:28.736459 15186 tablet_bootstrap.cc:654] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:28.736934 15186 log.cc:824] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.737689 15186 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: No bootstrap required, opened a new log
I20260619 20:52:28.737756 15186 ts_tablet_manager.cc:1397] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:28.737748 15188 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Bootstrap starting.
I20260619 20:52:28.738364 15188 tablet_bootstrap.cc:654] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:28.738896 15188 log.cc:824] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:28.739642 15188 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: No bootstrap required, opened a new log
I20260619 20:52:28.739715 15188 ts_tablet_manager.cc:1397] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:28.739871 15186 raft_consensus.cc:348] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.740042 15186 raft_consensus.cc:374] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:28.740077 15186 raft_consensus.cc:729] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4352ac050cce47b99f12220618a301b8, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.740229 15186 consensus_queue.cc:260] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.740512 15045 heartbeater.cc:503] Master 127.0.98.62:39435 was elected leader, sending a full tablet report...
I20260619 20:52:28.740763 15186 ts_tablet_manager.cc:1428] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:52:28.741691 15188 raft_consensus.cc:348] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.741816 15188 raft_consensus.cc:374] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:28.741837 15188 raft_consensus.cc:729] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc4e6142e0e4dc2a75774032bad566c, State: Initialized, Role: FOLLOWER
I20260619 20:52:28.741971 15188 consensus_queue.cc:260] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.742272 15188 ts_tablet_manager.cc:1428] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:52:28.742295 15177 heartbeater.cc:503] Master 127.0.98.62:39435 was elected leader, sending a full tablet report...
W20260619 20:52:28.832540 15046 tablet.cc:2369] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:52:28.888285 15190 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:28.888422 15190 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.888845 15190 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971), c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087)
I20260619 20:52:28.892830 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.889377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35675 (local address 127.0.98.1:46087)
0619 20:52:28.889444 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.889447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.889457 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:52:28.889506 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.889507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.889532 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.889603 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.889793 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.890501 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.891847 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.892468 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.892531 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.892591 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.892704 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.892744 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:52:28.892829 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.889221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40163 (local address 127.0.98.3:43971)
0619 20:52:28.889289 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.889315 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.889326 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:52:28.889357 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.889358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.889370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.889435 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:28.889677 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.890421 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.891847 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.892393 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.892471 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.892482 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.892713 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.892751 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:52:28.892851 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.889077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43971 (local address 127.0.98.2:40163)
0619 20:52:28.889163 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.889179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.889253 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.889492 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.889494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.889495 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.889592 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.889595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.890498 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.890499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.891754 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.891757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.892495 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.892496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.892614 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.892618 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.892687 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.892740 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":39}
I20260619 20:52:28.892858 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.889197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46087 (local address 127.0.98.2:35675)
0619 20:52:28.889406 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.889416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:28.889460 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:28.889644 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:28.889644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:28.889646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:28.889719 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.889720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.890570 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.890571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.891754 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:28.891757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.892554 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.892555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:28.892618 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.892621 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:52:28.892687 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:52:28.892740 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":10,"thread_start_us":155,"threads_started":1}
I20260619 20:52:28.893196 14861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432e1935054742b0a3513ec1bdd1f247" candidate_uuid: "4352ac050cce47b99f12220618a301b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a2dc262cef400490d7a853b0e4d062" is_pre_election: true
I20260619 20:52:28.893359 15131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432e1935054742b0a3513ec1bdd1f247" candidate_uuid: "4352ac050cce47b99f12220618a301b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc4e6142e0e4dc2a75774032bad566c" is_pre_election: true
I20260619 20:52:28.893407 14861 raft_consensus.cc:2454] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4352ac050cce47b99f12220618a301b8 in term 0.
I20260619 20:52:28.893491 15131 raft_consensus.cc:2454] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4352ac050cce47b99f12220618a301b8 in term 0.
I20260619 20:52:28.893651 14934 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4352ac050cce47b99f12220618a301b8, c5a2dc262cef400490d7a853b0e4d062; no voters:
I20260619 20:52:28.893774 15190 raft_consensus.cc:2781] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:52:28.893831 15190 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:52:28.893862 15190 raft_consensus.cc:3037] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:28.894593 15190 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.894723 15190 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [CANDIDATE]: Term 1 election: Requested vote from peers dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971), c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087)
I20260619 20:52:28.894858 15131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432e1935054742b0a3513ec1bdd1f247" candidate_uuid: "4352ac050cce47b99f12220618a301b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc4e6142e0e4dc2a75774032bad566c"
I20260619 20:52:28.894922 14861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432e1935054742b0a3513ec1bdd1f247" candidate_uuid: "4352ac050cce47b99f12220618a301b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a2dc262cef400490d7a853b0e4d062"
I20260619 20:52:28.894942 15131 raft_consensus.cc:3037] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:28.894996 14861 raft_consensus.cc:3037] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:28.895820 14861 raft_consensus.cc:2454] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4352ac050cce47b99f12220618a301b8 in term 1.
I20260619 20:52:28.895820 15131 raft_consensus.cc:2454] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4352ac050cce47b99f12220618a301b8 in term 1.
I20260619 20:52:28.895990 14934 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c; no voters:
I20260619 20:52:28.896087 15190 raft_consensus.cc:2781] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:28.896230 15190 raft_consensus.cc:686] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4352ac050cce47b99f12220618a301b8, State: Running, Role: LEADER
I20260619 20:52:28.896358 15190 consensus_queue.cc:237] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:28.897229 14718 catalog_manager.cc:5495] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4352ac050cce47b99f12220618a301b8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4352ac050cce47b99f12220618a301b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } health_report { overall_health: UNKNOWN } } }
I20260619 20:52:28.912370 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.910170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.0.98.2:35003)
0619 20:52:28.910236 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:28.910238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:28.910246 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:52:28.910316 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:28.910317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:28.910337 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:28.910399 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:52:28.910508 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.910978 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.911668 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:28.911852 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:28.911899 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:28.912020 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:52:28.912117 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:52:28.912176 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:28.912220 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:52:28.912238 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:52:28.914297 14861 raft_consensus.cc:1261] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Refusing update from remote peer 4352ac050cce47b99f12220618a301b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:28.914297 15131 raft_consensus.cc:1261] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Refusing update from remote peer 4352ac050cce47b99f12220618a301b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:28.914655 15193 consensus_queue.cc:1035] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:28.914788 15190 consensus_queue.cc:1035] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:28.919461 14718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:28.919555 14718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:28.920296 14718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Sending DeleteTablet for 3 replicas of tablet 432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:28.920574 14979 tablet_service.cc:1514] Processing DeleteTablet for tablet 432e1935054742b0a3513ec1bdd1f247 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260619 20:52:28.920679 15111 tablet_service.cc:1514] Processing DeleteTablet for tablet 432e1935054742b0a3513ec1bdd1f247 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36536
I20260619 20:52:28.920773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14757
I20260619 20:52:28.920774 14841 tablet_service.cc:1514] Processing DeleteTablet for tablet 432e1935054742b0a3513ec1bdd1f247 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33168
I20260619 20:52:28.920854 14916 tablet_replica.cc:330] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: stopping tablet replica
I20260619 20:52:28.920903 14916 raft_consensus.cc:2229] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:28.920958 14916 raft_consensus.cc:2258] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:28.921764 15206 tablet_replica.cc:330] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: stopping tablet replica
I20260619 20:52:28.921869 15206 raft_consensus.cc:2229] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:28.921976 15206 raft_consensus.cc:2258] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:28.922390 15206 ts_tablet_manager.cc:1905] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:28.924358 15206 ts_tablet_manager.cc:1918] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:28.924434 15206 log.cc:1196] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:28.924724 15206 ts_tablet_manager.cc:1939] T 432e1935054742b0a3513ec1bdd1f247 P dbc4e6142e0e4dc2a75774032bad566c: Deleting consensus metadata
I20260619 20:52:28.925102 14705 catalog_manager.cc:4841] TS dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): tablet 432e1935054742b0a3513ec1bdd1f247 (table test-table [id=e42e8820267e4509996e6dc2eb23ae92]) successfully deleted
I20260619 20:52:28.925446 15205 tablet_replica.cc:330] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: stopping tablet replica
I20260619 20:52:28.925547 15205 raft_consensus.cc:2229] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:52:28.925665 15205 raft_consensus.cc:2258] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:28.926043 15205 ts_tablet_manager.cc:1905] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:52:28.927738 14705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46087 (error 108)
I20260619 20:52:28.927758 15205 ts_tablet_manager.cc:1918] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:28.927817 15205 log.cc:1196] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:28.928110 15205 ts_tablet_manager.cc:1939] T 432e1935054742b0a3513ec1bdd1f247 P 4352ac050cce47b99f12220618a301b8: Deleting consensus metadata
I20260619 20:52:28.928340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14909
I20260619 20:52:28.929011 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.928434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46087 (local address 127.0.0.1:33178)
0619 20:52:28.928930 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.928983 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
W20260619 20:52:28.929140 14705 catalog_manager.cc:4584] TS c5a2dc262cef400490d7a853b0e4d062 (127.0.98.1:46087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 432e1935054742b0a3513ec1bdd1f247: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46087: connect: Connection refused (error 111)
I20260619 20:52:28.933648 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:28.933522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35003 (local address 127.0.0.1:59076)
0619 20:52:28.933579 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:28.933608 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:52:28.933735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 15048
I20260619 20:52:28.938251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 14686
I20260619 20:52:28.943527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39435
--webserver_interface=127.0.98.62
--webserver_port=42597
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:29.023823 15208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:29.024071 15208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:29.024122 15208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:29.025377 15208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:29.025434 15208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:29.025458 15208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:29.025487 15208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:29.025513 15208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:29.026947 15208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:29.027251 15208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:29.027475 15208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:29.032147 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.032159 15216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.032153 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:29.032450 15208 server_base.cc:1034] running on GCE node
I20260619 20:52:29.032682 15208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:29.033018 15208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:29.034179 15208 hybrid_clock.cc:648] HybridClock initialized: now 1781902349034170 us; error 32 us; skew 500 ppm
Jun 19 20:52:29 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:29.036634 15208 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:29.037046 15208 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260619 20:52:29.037269 15208 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:29.037386 15208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:29.039176 15208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:52:29.040064 15223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.040234 15208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:29.040313 15208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7caed2c800e749c483831f48f96a7c54"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "b8c09e6f23357a6cd1f6525d3c736aab"
server_key_iv: "072e97560c0bd461e0df000f03502e3e"
server_key_version: "encryptionkey@0"
I20260619 20:52:29.040733 15208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:29.055239 15208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:29.056946 15208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:29.057090 15208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:29.060828 15208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39435
I20260619 20:52:29.060875 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39435 every 8 connection(s)
I20260619 20:52:29.061257 15208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:29.062798 15276 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:52:29.063445 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap starting.
I20260619 20:52:29.068322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15208
I20260619 20:52:29.068651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46087
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39435
--webserver_port=34385
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:29.072072 15276 log.cc:824] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:29.074302 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:52:29.074589 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54: Bootstrap complete.
I20260619 20:52:29.076747 15276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:29.076983 15276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caed2c800e749c483831f48f96a7c54, State: Initialized, Role: FOLLOWER
I20260619 20:52:29.077170 15276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:29.077248 15276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:29.077316 15276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:29.077371 15276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:52:29.078325 15276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:29.078444 15276 leader_election.cc:302] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caed2c800e749c483831f48f96a7c54; no voters:
I20260619 20:52:29.078642 15276 leader_election.cc:288] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 20:52:29.078675 15280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:52:29.078785 15280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [term 3 LEADER]: Becoming Leader. State: Replica: 7caed2c800e749c483831f48f96a7c54, State: Running, Role: LEADER
I20260619 20:52:29.078872 15280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } }
I20260619 20:52:29.078919 15276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:29.079126 15280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7caed2c800e749c483831f48f96a7c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } } }
I20260619 20:52:29.079192 15280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:29.079312 15280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caed2c800e749c483831f48f96a7c54. Latest consensus state: current_term: 3 leader_uuid: "7caed2c800e749c483831f48f96a7c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caed2c800e749c483831f48f96a7c54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39435 } } }
I20260619 20:52:29.079378 15280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caed2c800e749c483831f48f96a7c54 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:29.079444 15284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:29.080240 15284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:29.080531 15284 catalog_manager.cc:1259] Loaded cluster ID: fa454e53c2c2487f981dcf27241106a8
I20260619 20:52:29.080592 15284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 20:52:29.080806 15284 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbe620109ad google::LogMessage::Fail() at ??:0
@ 0x7fbe6201388a google::LogMessage::SendToLog() at ??:0
@ 0x7fbe620104de google::LogMessage::Flush() at ??:0
@ 0x7fbe62011a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe69f310f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbe69f3a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbe63ede0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe63ed3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe64d376db start_thread at ??:0
@ 0x7fbe60b0e61f clone at ??:0
W20260619 20:52:29.165573 15278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:29.165769 15278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:29.165793 15278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:29.166977 15278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:29.167019 15278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:29.167045 15278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:52:29.168495 15278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34385
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:29.168743 15278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:29.168987 15278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:29.174019 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.174026 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.174093 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:29.174337 15278 server_base.cc:1034] running on GCE node
I20260619 20:52:29.174559 15278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:29.174824 15278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:29.175979 15278 hybrid_clock.cc:648] HybridClock initialized: now 1781902349175945 us; error 45 us; skew 500 ppm
Jun 19 20:52:29 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:29.178726 15278 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:52:29.179193 15278 webserver.cc:460] Webserver started at http://127.0.98.1:34385/ using document root <none> and password file <none>
I20260619 20:52:29.179450 15278 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:29.179497 15278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:29.181066 15278 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:52:29.181831 15322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.182044 15278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:52:29.182143 15278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "c5a2dc262cef400490d7a853b0e4d062"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "6d5eb4a63a64322f38be4541dd5a2760"
server_key_iv: "901a7e5b92c380c46bd8be8c3c456a6b"
server_key_version: "encryptionkey@0"
I20260619 20:52:29.182523 15278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:29.200613 15278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:29.203209 15278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:29.203377 15278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:29.203622 15278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:29.204129 15329 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:52:29.206382 15278 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:52:29.206452 15278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:52:29.206496 15278 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:52:29.207144 15278 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:52:29.207199 15278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:52:29.207273 15329 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap starting.
I20260619 20:52:29.214535 15278 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46087
I20260619 20:52:29.214972 15278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:52:29.217494 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46087 every 8 connection(s)
I20260619 20:52:29.218194 15329 log.cc:824] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:29.220844 15329 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:52:29.221082 15329 tablet_bootstrap.cc:492] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Bootstrap complete.
I20260619 20:52:29.222286 15329 ts_tablet_manager.cc:1397] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260619 20:52:29.223824 15329 raft_consensus.cc:348] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:29.224176 15329 raft_consensus.cc:729] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a2dc262cef400490d7a853b0e4d062, State: Initialized, Role: FOLLOWER
I20260619 20:52:29.224291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15278
I20260619 20:52:29.224336 15329 consensus_queue.cc:260] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:29.224591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35003
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39435
--webserver_port=40969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:52:29.224617 15329 ts_tablet_manager.cc:1428] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260619 20:52:29.281325 15439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:52:29.215712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:55479)
0619 20:52:29.215925 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.215961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:29.216420 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:29.281079 (+ 64659us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":127,"threads_started":1}
W20260619 20:52:29.281589 15437 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39435 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39435: Client connection negotiation failed: client connection to 127.0.98.62:39435: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:52:29.281967 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.281838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:44957)
0619 20:52:29.281899 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.281943 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 20:52:29.282323 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.282220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:36587)
0619 20:52:29.282270 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.282300 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:52:29.282447 15437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:52:29.308696 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:29.308914 15442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:29.308939 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:29.310215 15442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:29.310261 15442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:29.310288 15442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:52:29.311757 15442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40969
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:29.312003 15442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:29.312255 15442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:29.317471 15450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:29.317625 15442 server_base.cc:1034] running on GCE node
W20260619 20:52:29.317497 15447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.317703 15448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:29.317916 15442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:29.318205 15442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:29.319345 15442 hybrid_clock.cc:648] HybridClock initialized: now 1781902349319336 us; error 30 us; skew 500 ppm
Jun 19 20:52:29 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:29.321879 15442 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:52:29.322324 15442 webserver.cc:460] Webserver started at http://127.0.98.2:40969/ using document root <none> and password file <none>
I20260619 20:52:29.322502 15442 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:29.322542 15442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:29.323820 15442 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:29.324412 15457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.324618 15442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:52:29.324725 15442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "4352ac050cce47b99f12220618a301b8"
format_stamp: "Formatted at 2026-06-19 20:52:27 on dist-test-slave-0smx"
server_key: "2f1db14b894edc3aa08abf1d5dcf80a4"
server_key_iv: "c9fda8cc8ca1ca12ba7e25ae28cf7b25"
server_key_version: "encryptionkey@0"
I20260619 20:52:29.325112 15442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:29.357973 15442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:29.360162 15442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:29.360343 15442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:29.360607 15442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:29.360939 15442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:29.360996 15442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.361044 15442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:29.361083 15442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.367071 15442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35003
I20260619 20:52:29.367118 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35003 every 8 connection(s)
I20260619 20:52:29.367460 15442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:52:29.368531 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.368156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:45493)
0619 20:52:29.368323 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.368392 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260619 20:52:29.368943 15571 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39435 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39435: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
I20260619 20:52:29.369246 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.369148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:38047)
0619 20:52:29.369199 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.369223 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:52:29.369539 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.369456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:40061)
0619 20:52:29.369499 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.369521 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:52:29.369650 15571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:52:29.370174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15442
I20260619 20:52:29.370432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39435
--webserver_port=37375
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:29.451586 15574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:29.451788 15574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:29.451812 15574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:29.452877 15574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:29.452912 15574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:29.452936 15574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:52:29.454293 15574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37375
--tserver_master_addrs=127.0.98.62:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:29.454459 15574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:29.454695 15574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:29.459774 15580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.459790 15582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:29.459762 15579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:29.459857 15574 server_base.cc:1034] running on GCE node
I20260619 20:52:29.460204 15574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:29.460449 15574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:29.461640 15574 hybrid_clock.cc:648] HybridClock initialized: now 1781902349461622 us; error 47 us; skew 500 ppm
Jun 19 20:52:29 dist-test-slave-0smx krb5kdc[14138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:29.464280 15574 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:52:29.464671 15574 webserver.cc:460] Webserver started at http://127.0.98.3:37375/ using document root <none> and password file <none>
I20260619 20:52:29.464924 15574 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:29.465009 15574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:29.466444 15574 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:29.467042 15589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.467211 15574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:29.467301 15574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "dbc4e6142e0e4dc2a75774032bad566c"
format_stamp: "Formatted at 2026-06-19 20:52:28 on dist-test-slave-0smx"
server_key: "fb0f9b681c9cd6fb44dcb48b22646446"
server_key_iv: "53845c06ea864c4cc793d076e81ab261"
server_key_version: "encryptionkey@0"
I20260619 20:52:29.467674 15574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:29.484431 15574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:29.486912 15574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:29.487089 15574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:29.487345 15574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:29.487659 15574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:29.487712 15574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.487752 15574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:29.487795 15574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:29.493695 15574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43971
I20260619 20:52:29.493799 15702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43971 every 8 connection(s)
I20260619 20:52:29.494138 15574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:52:29.495235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15574
I20260619 20:52:29.495257 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.494850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:43279)
0619 20:52:29.495013 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.495120 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
W20260619 20:52:29.495821 15703 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39435 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39435: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
I20260619 20:52:29.496109 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.496021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:53939)
0619 20:52:29.496066 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.496089 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:52:29.496409 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:29.496334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:55683)
0619 20:52:29.496376 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:29.496396 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:52:29.496553 15703 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:52:30.283130 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.282885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:49391)
0619 20:52:30.283038 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:30.283071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260619 20:52:30.370175 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.369982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:52279)
0619 20:52:30.370097 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:30.370128 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260619 20:52:30.497020 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.496828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:34483)
0619 20:52:30.496940 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:30.496972 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260619 20:52:30.590662 15710 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:30.590840 15710 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:30.591190 15710 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
Jun 19 20:52:30 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 20:52:30 dist-test-slave-0smx krb5kdc[14138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 20:52:30.601792 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.591380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35003 (local address 127.0.98.1:60547)
0619 20:52:30.591511 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:30.591520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:30.591558 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:30.592369 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:30.592372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:30.592767 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:30.592948 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:30.592953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.594096 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.594097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:30.594859 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:30.594861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.595456 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.595457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:30.595570 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:30.596196 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:30.596205 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:30.598199 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:30.600336 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:30.600337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:30.600563 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:30.600829 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:30.600830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:30.600886 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:30.601345 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:30.601346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:30.601396 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:52:30.601494 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 20:52:30.601682 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":16,"thread_start_us":61,"threads_started":1}
I20260619 20:52:30.601840 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.591497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60547 (local address 127.0.98.2:35003)
0619 20:52:30.591663 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:30.591676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:30.591704 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:52:30.592150 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:30.592152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:30.592188 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:30.592352 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:30.593070 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.593993 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.594974 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.595351 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.595480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:30.598301 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:30.598305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:30.598305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:30.598317 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:30.600159 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600647 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:30.600650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:30.600650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:30.600756 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600978 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:30.600979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:30.600980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:30.601244 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:30.601343 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:30.601575 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 20:52:30.601726 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":57,"threads_started":1}
I20260619 20:52:30.602232 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.591497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45187 (local address 127.0.98.3:43971)
0619 20:52:30.591678 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:30.591686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:30.591711 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:52:30.592145 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:30.592147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:30.592188 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:30.592341 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:30.593080 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.593986 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.594961 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.595356 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.595481 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:30.597928 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:30.597931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:30.597931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:30.597942 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:30.599852 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600345 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:30.600349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:30.600349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:30.600458 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600754 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:30.600755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:30.600755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:30.601116 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:30.601204 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:30.601998 (+ 794us) server_negotiation.cc:299] Negotiation successful
0619 20:52:30.602129 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":54,"threads_started":1}
I20260619 20:52:30.602097 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:30.591369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43971 (local address 127.0.98.1:45187)
0619 20:52:30.591424 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:30.591435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:30.591490 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:30.592368 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:30.592372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:30.592766 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:30.592948 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:30.592953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.594125 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.594128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:30.594861 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:30.594863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:30.595457 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:30.595458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:30.595576 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:30.596217 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:30.596222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:30.597795 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:30.599935 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:30.599941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:30.599943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:30.600248 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:30.600539 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:30.600540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:30.600540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:30.600671 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:30.601195 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:30.601201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:30.601368 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 20:52:30.601509 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 20:52:30.601797 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":177}
W20260619 20:52:30.602664 15323 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
W20260619 20:52:30.603065 15325 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:30.603106 15325 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a2dc262cef400490d7a853b0e4d062; no voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c
I20260619 20:52:30.603230 15710 raft_consensus.cc:2726] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:52:31.283785 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:31.283568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:38991)
0619 20:52:31.283701 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:31.283730 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260619 20:52:31.370795 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:31.370602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:41979)
0619 20:52:31.370727 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:31.370760 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260619 20:52:31.497720 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:31.497508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:42241)
0619 20:52:31.497649 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:31.497683 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260619 20:52:32.284574 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:32.284350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:51839)
0619 20:52:32.284483 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:32.284522 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260619 20:52:32.371659 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:32.371393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:55615)
0619 20:52:32.371581 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:32.371619 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260619 20:52:32.498404 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:32.498180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:41819)
0619 20:52:32.498320 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:32.498359 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260619 20:52:32.507974 15720 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:32.508064 15720 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:32.508225 15720 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
W20260619 20:52:32.508613 15325 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
W20260619 20:52:32.508706 15323 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:32.508807 15323 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a2dc262cef400490d7a853b0e4d062; no voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c
I20260619 20:52:32.508914 15720 raft_consensus.cc:2726] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:52:33.285498 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:33.285219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:59137)
0619 20:52:33.285413 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:33.285449 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":130,"threads_started":1}
I20260619 20:52:33.372347 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:33.372120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:55337)
0619 20:52:33.372269 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:33.372307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260619 20:52:33.499181 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:33.498921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:43499)
0619 20:52:33.499093 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:33.499138 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260619 20:52:34.286173 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:34.285950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:56825)
0619 20:52:34.286090 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:34.286125 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260619 20:52:34.373052 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:34.372843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:44641)
0619 20:52:34.372965 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:34.373008 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260619 20:52:34.499893 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:34.499656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:35117)
0619 20:52:34.499796 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:34.499839 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260619 20:52:34.886247 15727 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:34.886385 15727 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:34.886593 15727 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
W20260619 20:52:34.887105 15325 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
W20260619 20:52:34.887178 15323 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:34.887236 15323 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a2dc262cef400490d7a853b0e4d062; no voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c
I20260619 20:52:34.887307 15727 raft_consensus.cc:2726] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:52:35.286877 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:35.286652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:33415)
0619 20:52:35.286785 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:35.286824 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260619 20:52:35.373718 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:35.373519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:56707)
0619 20:52:35.373638 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:35.373677 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260619 20:52:35.500672 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:35.500389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:36299)
0619 20:52:35.500543 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:35.500581 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260619 20:52:36.287657 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:36.287308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:59245)
0619 20:52:36.287575 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:36.287609 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
I20260619 20:52:36.374426 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:36.374153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:42651)
0619 20:52:36.374346 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:36.374385 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260619 20:52:36.501405 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:36.501123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:47545)
0619 20:52:36.501245 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:36.501284 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260619 20:52:37.288287 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:37.288089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:55895)
0619 20:52:37.288205 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:37.288241 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260619 20:52:37.375111 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:37.374896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:43397)
0619 20:52:37.375029 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:37.375068 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260619 20:52:37.502161 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:37.501950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:36607)
0619 20:52:37.502083 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:37.502120 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260619 20:52:38.288916 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:38.288717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:51625)
0619 20:52:38.288834 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:38.288868 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260619 20:52:38.375919 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:38.375678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:56365)
0619 20:52:38.375839 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:38.375877 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260619 20:52:38.502806 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:38.502598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:49349)
0619 20:52:38.502725 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:38.502765 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260619 20:52:39.011740 15740 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:39.011826 15740 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:39.011992 15740 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
W20260619 20:52:39.012396 15325 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
W20260619 20:52:39.012482 15323 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:39.012566 15323 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a2dc262cef400490d7a853b0e4d062; no voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c
I20260619 20:52:39.012652 15740 raft_consensus.cc:2726] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:52:39.289649 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:39.289435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:50757)
0619 20:52:39.289566 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:39.289602 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260619 20:52:39.376574 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:39.376363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:49399)
0619 20:52:39.376495 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:39.376534 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260619 20:52:39.503432 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:39.503229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:52891)
0619 20:52:39.503357 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:39.503389 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260619 20:52:40.290424 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:40.290181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:59211)
0619 20:52:40.290331 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:40.290376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260619 20:52:40.377408 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:40.377107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:39761)
0619 20:52:40.377245 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:40.377347 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260619 20:52:40.504146 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:40.503895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:40259)
0619 20:52:40.504053 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:40.504100 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260619 20:52:41.291100 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:41.290890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:38041)
0619 20:52:41.291014 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:41.291052 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260619 20:52:41.378098 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:41.377861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:41887)
0619 20:52:41.378017 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:41.378058 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260619 20:52:41.504853 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:41.504647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:59655)
0619 20:52:41.504778 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:41.504808 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260619 20:52:42.291746 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:42.291544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:44503)
0619 20:52:42.291667 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:42.291700 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260619 20:52:42.378825 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:42.378621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:60641)
0619 20:52:42.378746 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:42.378783 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260619 20:52:42.505556 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:42.505280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:51365)
0619 20:52:42.505459 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:42.505498 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260619 20:52:43.204869 15753 raft_consensus.cc:482] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:43.204988 15753 raft_consensus.cc:504] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4352ac050cce47b99f12220618a301b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35003 } } peers { permanent_uuid: "dbc4e6142e0e4dc2a75774032bad566c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43971 } } peers { permanent_uuid: "c5a2dc262cef400490d7a853b0e4d062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46087 } }
I20260619 20:52:43.205200 15753 leader_election.cc:288] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003), dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971)
W20260619 20:52:43.205590 15323 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4352ac050cce47b99f12220618a301b8 (127.0.98.2:35003): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
W20260619 20:52:43.205691 15325 leader_election.cc:341] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbc4e6142e0e4dc2a75774032bad566c (127.0.98.3:43971): Not found: Tablet not found: 432e1935054742b0a3513ec1bdd1f247
I20260619 20:52:43.205777 15325 leader_election.cc:302] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a2dc262cef400490d7a853b0e4d062; no voters: 4352ac050cce47b99f12220618a301b8, dbc4e6142e0e4dc2a75774032bad566c
I20260619 20:52:43.205865 15753 raft_consensus.cc:2726] T 432e1935054742b0a3513ec1bdd1f247 P c5a2dc262cef400490d7a853b0e4d062 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:52:43.292446 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:43.292225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:49995)
0619 20:52:43.292355 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:43.292394 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260619 20:52:43.379505 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:43.379273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:40885)
0619 20:52:43.379418 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:43.379445 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260619 20:52:43.506183 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:43.505988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:46565)
0619 20:52:43.506112 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:43.506142 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260619 20:52:44.293216 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:44.292980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.1:54813)
0619 20:52:44.293128 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:44.293169 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260619 20:52:44.380318 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:44.380062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.2:58125)
0619 20:52:44.380232 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:44.380274 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260619 20:52:44.503675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 15278
I20260619 20:52:44.506884 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:44.506657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39435 (local address 127.0.98.3:48867)
0619 20:52:44.506788 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:44.506827 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"spinlock_wait_cycles":15104,"thread_start_us":109,"threads_started":1}
I20260619 20:52:44.508545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 15442
I20260619 20:52:44.512792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 15574
2026-06-19T20:52:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18218 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:52:44 dist-test-slave-0smx krb5kdc[15762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:52:44 dist-test-slave-0smx krb5kdc[15762](info): set up 2 sockets
Jun 19 20:52:44 dist-test-slave-0smx krb5kdc[15762](info): commencing operation
krb5kdc: starting...
W20260619 20:52:45.553491 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:52:45 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:52:45.666080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:45.745501 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:45.745692 15775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:45.745713 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:45.746855 15775 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:52:45.746894 15775 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:52:45.746908 15775 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:52:45.746919 15775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:52:45.746970 15775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:45.746989 15775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:45.747012 15775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:52:45.747040 15775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:52:45.748473 15775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:45.748692 15775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:45.748967 15775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:45.754022 15783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:45.754051 15775 server_base.cc:1034] running on GCE node
W20260619 20:52:45.754050 15780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:45.754024 15781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:45.754534 15775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:45.754844 15775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:45.756003 15775 hybrid_clock.cc:648] HybridClock initialized: now 1781902365755998 us; error 35 us; skew 500 ppm
Jun 19 20:52:45 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:45.758342 15775 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:52:45.758764 15775 webserver.cc:460] Webserver started at http://127.0.98.62:43349/ using document root <none> and password file <none>
I20260619 20:52:45.758991 15775 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:45.759068 15775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:45.759218 15775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:45.760255 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "48f4db8e2e4c46c693e294784fc1a115"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "c31c809ade349cc2f6d0b7cc05558908"
server_key_iv: "0e11a92e77cb0694f08233997d94ab83"
server_key_version: "encryptionkey@0"
I20260619 20:52:45.760603 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "48f4db8e2e4c46c693e294784fc1a115"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "c31c809ade349cc2f6d0b7cc05558908"
server_key_iv: "0e11a92e77cb0694f08233997d94ab83"
server_key_version: "encryptionkey@0"
I20260619 20:52:45.761969 15775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:52:45.762645 15790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:45.762822 15775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:45.762924 15775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "48f4db8e2e4c46c693e294784fc1a115"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "c31c809ade349cc2f6d0b7cc05558908"
server_key_iv: "0e11a92e77cb0694f08233997d94ab83"
server_key_version: "encryptionkey@0"
I20260619 20:52:45.762995 15775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:45.775415 15775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:45.777052 15775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:45.777200 15775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:45.780763 15775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34465
I20260619 20:52:45.780846 15842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34465 every 8 connection(s)
I20260619 20:52:45.781173 15775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:52:45.782042 15843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:45.784163 15843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap starting.
I20260619 20:52:45.784714 15843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:45.784994 15843 log.cc:824] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:45.785684 15843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: No bootstrap required, opened a new log
I20260619 20:52:45.787139 15843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:52:45.787271 15843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:45.787317 15843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f4db8e2e4c46c693e294784fc1a115, State: Initialized, Role: FOLLOWER
I20260619 20:52:45.787456 15843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:52:45.787536 15843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:52:45.787585 15843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:52:45.787631 15843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:45.788110 15843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:52:45.788211 15843 leader_election.cc:302] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f4db8e2e4c46c693e294784fc1a115; no voters:
I20260619 20:52:45.788389 15843 leader_election.cc:288] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:52:45.788455 15846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:45.788611 15843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:52:45.788676 15846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 LEADER]: Becoming Leader. State: Replica: 48f4db8e2e4c46c693e294784fc1a115, State: Running, Role: LEADER
I20260619 20:52:45.788815 15846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:52:45.789172 15847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f4db8e2e4c46c693e294784fc1a115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } } }
I20260619 20:52:45.789247 15847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:45.789450 15848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f4db8e2e4c46c693e294784fc1a115. Latest consensus state: current_term: 1 leader_uuid: "48f4db8e2e4c46c693e294784fc1a115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } } }
I20260619 20:52:45.789597 15854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:52:45.789762 15848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: This master's current role is: LEADER
I20260619 20:52:45.790185 15854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:52:45.790382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15775
I20260619 20:52:45.790468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:52:45.790655 392 external_mini_cluster.cc:1428] Setting key e936aab0f41eb6e8dcfa9de62f7fa322
I20260619 20:52:45.791754 15854 catalog_manager.cc:1347] Generated new cluster ID: 5a2389d00e9b448ebf411e3f01aac3c5
I20260619 20:52:45.791812 15854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:52:45.797180 15854 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:52:45.797881 15854 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 20:52:45 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:45.802172 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:45.791089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.0.98.62:34465)
0619 20:52:45.791452 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:45.791461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:45.791495 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:52:45.791603 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:45.791610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:45.791662 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:45.791794 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:45.792561 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:45.793900 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:45.795007 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:45.795616 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:45.795707 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:45.798525 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:45.798528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:45.798529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:45.798540 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:45.800418 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:45.800774 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:45.800776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:45.800777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:45.800863 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:45.801034 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:45.801034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:45.801035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:45.801275 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:45.801444 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:45.801820 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 20:52:45.801953 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":72,"threads_started":1}
I20260619 20:52:45.804966 15854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Generated new TSK 0
I20260619 20:52:45.805125 15854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:52:45.835655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34465
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:45.918025 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:45.918229 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:45.918255 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:45.919402 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:45.919448 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:45.919504 15869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:52:45.921484 15869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:45.921728 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:45.921979 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:52:45.926987 15869 server_base.cc:1034] running on GCE node
W20260619 20:52:45.926928 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:45.926975 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:45.926928 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:45.927336 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:45.927637 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:45.928819 15869 hybrid_clock.cc:648] HybridClock initialized: now 1781902365928791 us; error 39 us; skew 500 ppm
Jun 19 20:52:45 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:45.931584 15869 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:52:45.932145 15869 webserver.cc:460] Webserver started at http://127.0.98.1:35207/ using document root <none> and password file <none>
I20260619 20:52:45.932382 15869 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:45.932428 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:45.932586 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:45.933790 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "559c5f912fc6477b985ea848568fb7cc"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "e8c3de60401653cd786aada9e0473712"
server_key_iv: "f2fe58eb80f444292d2f78543d8eddc3"
server_key_version: "encryptionkey@0"
I20260619 20:52:45.934170 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "559c5f912fc6477b985ea848568fb7cc"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "e8c3de60401653cd786aada9e0473712"
server_key_iv: "f2fe58eb80f444292d2f78543d8eddc3"
server_key_version: "encryptionkey@0"
I20260619 20:52:45.935521 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:52:45.936259 15884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:45.936440 15869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:45.936542 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "559c5f912fc6477b985ea848568fb7cc"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "e8c3de60401653cd786aada9e0473712"
server_key_iv: "f2fe58eb80f444292d2f78543d8eddc3"
server_key_version: "encryptionkey@0"
I20260619 20:52:45.936645 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:45.969805 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:45.972435 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:45.972620 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:45.972888 15869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:45.973232 15869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:45.973289 15869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:45.973345 15869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:45.973397 15869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:45.979702 15869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39281
I20260619 20:52:45.979745 15997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39281 every 8 connection(s)
I20260619 20:52:45.980126 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:52:45.980986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 15869
I20260619 20:52:45.981109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:52:45.981406 392 external_mini_cluster.cc:1428] Setting key c2e9f44a6a3c79e752408783ca6d1d38
Jun 19 20:52:45 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:52:45.991138 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:45.980913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54973 (local address 127.0.98.62:34465)
0619 20:52:45.980997 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:45.980999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:45.981216 (+ 217us) server_negotiation.cc:407] Connection header received
0619 20:52:45.981997 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:45.981999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:45.982009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:45.982071 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:45.983130 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:45.983812 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:45.985090 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:45.985476 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:45.985556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:45.988081 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:45.988082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:45.988082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:45.988091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:45.989813 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:45.990194 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:45.990195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:45.990196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:45.990261 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:45.990505 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:45.990505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:45.990506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:45.990606 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:45.990656 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:45.990998 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 20:52:45.991071 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 20:52:45.991321 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:45.980880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:54973)
0619 20:52:45.981076 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:45.981130 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:45.981897 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:45.982150 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:45.982155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:45.982784 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:45.983016 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:45.983021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:45.983909 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:45.983910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:45.984994 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:45.984996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:45.985579 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:45.985581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:45.985744 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:45.986301 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:45.986309 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:45.987981 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:45.989910 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:45.989915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:45.989917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:45.990143 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:45.990345 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:45.990346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:45.990346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:45.990450 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:45.990689 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:45.990693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:45.990875 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 20:52:45.990970 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 20:52:45.991088 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260619 20:52:45.991868 15998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34465
I20260619 20:52:45.991961 15998 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:45.992095 15998 heartbeater.cc:511] Master 127.0.98.62:34465 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 20:52:45.992625 15807 ts_manager.cc:194] Registered new tserver with Master: 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281)
I20260619 20:52:45.993212 15807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:52:46.006181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34465
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:46.087670 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:46.087903 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:46.087927 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:46.089208 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:46.089358 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:46.089401 16003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:52:46.091001 16003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:46.091182 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:46.091369 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:46.096406 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:46.096509 16003 server_base.cc:1034] running on GCE node
W20260619 20:52:46.096412 16009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:46.096647 16008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:46.096895 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:46.097178 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:46.098326 16003 hybrid_clock.cc:648] HybridClock initialized: now 1781902366098310 us; error 54 us; skew 500 ppm
Jun 19 20:52:46 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:46.101066 16003 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:52:46.101533 16003 webserver.cc:460] Webserver started at http://127.0.98.2:46411/ using document root <none> and password file <none>
I20260619 20:52:46.101711 16003 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:46.101756 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:46.101842 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:46.102805 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "445ff1991f1c4d5891c8ac77c544ca61"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "77f291f52b2bf1ee55c8673aa19575b5"
server_key_iv: "8547f27e137d42150e927346f0933ccb"
server_key_version: "encryptionkey@0"
I20260619 20:52:46.103093 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "445ff1991f1c4d5891c8ac77c544ca61"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "77f291f52b2bf1ee55c8673aa19575b5"
server_key_iv: "8547f27e137d42150e927346f0933ccb"
server_key_version: "encryptionkey@0"
I20260619 20:52:46.104310 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:52:46.104962 16018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:46.105140 16003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:52:46.105233 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "445ff1991f1c4d5891c8ac77c544ca61"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "77f291f52b2bf1ee55c8673aa19575b5"
server_key_iv: "8547f27e137d42150e927346f0933ccb"
server_key_version: "encryptionkey@0"
I20260619 20:52:46.105332 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:46.114193 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:46.116410 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:46.116560 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:46.116813 16003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:46.117122 16003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:46.117175 16003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:46.117215 16003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:46.117242 16003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:46.123070 16003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36109
I20260619 20:52:46.123121 16131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36109 every 8 connection(s)
I20260619 20:52:46.123448 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:52:46 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:46.130514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16003
I20260619 20:52:46.130607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:52:46.130807 392 external_mini_cluster.cc:1428] Setting key 5dd8bbdf0101dbc47fe24d108bbf5f9f
I20260619 20:52:46.132951 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.124142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45233 (local address 127.0.98.62:34465)
0619 20:52:46.124229 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.124232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.124429 (+ 197us) server_negotiation.cc:407] Connection header received
0619 20:52:46.124890 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.124891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.124900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.124952 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:46.125910 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.126399 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.127318 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.127496 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.127547 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.129661 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:46.129662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:46.129662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:46.129670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:46.131292 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:46.131707 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:46.131708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:46.131708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:46.131768 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:46.132080 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:46.132081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:46.132081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:46.132259 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:46.132331 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.132725 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.132821 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 20:52:46.133286 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.124123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:45233)
0619 20:52:46.124307 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.124343 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.124809 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.125004 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.125007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.125623 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:46.125830 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.125835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.126477 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.126478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.127244 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.127246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.127568 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.127569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.127663 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.128201 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:46.128209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:46.129550 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:46.131368 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:46.131375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:46.131377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:46.131636 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:46.131839 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:46.131840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:46.131840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:46.131983 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:46.132343 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:46.132349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:46.132516 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.132629 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.132858 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260619 20:52:46.133814 16132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34465
I20260619 20:52:46.133927 16132 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:46.134133 16132 heartbeater.cc:511] Master 127.0.98.62:34465 requested a full tablet report, sending...
I20260619 20:52:46.134521 15807 ts_manager.cc:194] Registered new tserver with Master: 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
I20260619 20:52:46.134917 15807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45233
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:52:46.155287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34465
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:52:46.235659 16137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:52:46.235891 16137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:52:46.235927 16137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:52:46.237073 16137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:52:46.237138 16137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:52:46.237181 16137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:52:46.238649 16137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:52:46.238919 16137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:52:46.239142 16137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:52:46.244166 16145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:46.244180 16142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:52:46.244182 16143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:52:46.244419 16137 server_base.cc:1034] running on GCE node
I20260619 20:52:46.244660 16137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:52:46.244895 16137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:52:46.246078 16137 hybrid_clock.cc:648] HybridClock initialized: now 1781902366246066 us; error 38 us; skew 500 ppm
Jun 19 20:52:46 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:52:46.248544 16137 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:52:46.249059 16137 webserver.cc:460] Webserver started at http://127.0.98.3:40197/ using document root <none> and password file <none>
I20260619 20:52:46.249346 16137 fs_manager.cc:359] Metadata directory not provided
I20260619 20:52:46.249424 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:52:46.249562 16137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:52:46.250734 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "89f5de920cdb4872b35dad447402a3dc"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "f02c19d73151fee0b2e3a1aa7cdd8389"
server_key_iv: "dc996e7a70751b27fc5010cfd2b0761d"
server_key_version: "encryptionkey@0"
I20260619 20:52:46.251096 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "89f5de920cdb4872b35dad447402a3dc"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "f02c19d73151fee0b2e3a1aa7cdd8389"
server_key_iv: "dc996e7a70751b27fc5010cfd2b0761d"
server_key_version: "encryptionkey@0"
I20260619 20:52:46.252483 16137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:52:46.253484 16152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:46.253630 16137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:52:46.253715 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "89f5de920cdb4872b35dad447402a3dc"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "f02c19d73151fee0b2e3a1aa7cdd8389"
server_key_iv: "dc996e7a70751b27fc5010cfd2b0761d"
server_key_version: "encryptionkey@0"
I20260619 20:52:46.253777 16137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:52:46.275581 16137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:52:46.277383 16137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:52:46.277509 16137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:52:46.277700 16137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:52:46.278013 16137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:52:46.278044 16137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:46.278086 16137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:52:46.278112 16137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:52:46.283752 16137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34695
I20260619 20:52:46.283826 16265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34695 every 8 connection(s)
I20260619 20:52:46.284168 16137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:52:46.289851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16137
I20260619 20:52:46.289958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:52:46.290126 392 external_mini_cluster.cc:1428] Setting key da0633fd1b7bd4ca98c98b8056f7a9a3
Jun 19 20:52:46 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:52:46.293542 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.284874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52069 (local address 127.0.98.62:34465)
0619 20:52:46.284953 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.284955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.285141 (+ 186us) server_negotiation.cc:407] Connection header received
0619 20:52:46.285702 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.285703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.285713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.285766 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:46.286696 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.287209 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.288179 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.288379 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.288443 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.291087 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:46.291088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:46.291089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:46.291095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:46.292294 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:46.292649 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:46.292650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:46.292651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:46.292696 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:46.292975 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:46.292976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:46.292976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:46.293072 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:46.293123 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.293405 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.293469 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:52:46.293769 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.284823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:52069)
0619 20:52:46.284995 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.285030 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.285607 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.285835 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.285838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.286398 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:52:46.286582 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.286587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.287285 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.287286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.288084 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.288086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.288460 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.288461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.288598 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.289150 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:52:46.289158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:52:46.290995 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:52:46.292365 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:46.292370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:46.292372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:46.292584 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:46.292754 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:52:46.292755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:52:46.292755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:52:46.292911 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:52:46.293128 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:52:46.293132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:52:46.293240 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.293388 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.293506 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260619 20:52:46.294306 16266 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34465
I20260619 20:52:46.294402 16266 heartbeater.cc:464] Registering TS with master...
I20260619 20:52:46.294582 16266 heartbeater.cc:511] Master 127.0.98.62:34465 requested a full tablet report, sending...
I20260619 20:52:46.294929 15807 ts_manager.cc:194] Registered new tserver with Master: 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695)
I20260619 20:52:46.295248 15807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52069
I20260619 20:52:46.302054 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:52:46.309056 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.303292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.62:34465)
0619 20:52:46.303350 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.303352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.303486 (+ 134us) server_negotiation.cc:407] Connection header received
0619 20:52:46.303522 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.303523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.303530 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.303568 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:52:46.304068 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.304521 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.305287 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.305570 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.305630 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.306780 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:52:46.306781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:52:46.306781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:52:46.306787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:52:46.308001 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:46.308330 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:46.308330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:46.308331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:46.308376 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:52:46.308538 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:52:46.308538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:52:46.308538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:52:46.308662 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:52:46.308744 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.308910 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.308967 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:52:46.310426 15807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:52:46.311964 15807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:52:46.318413 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.314294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34695 (local address 127.0.0.1:39244)
0619 20:52:46.314384 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.314423 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.314481 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.315024 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.315027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.315030 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:46.315103 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.315105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.316050 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.316052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.317223 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.317225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.318145 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.318146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.318221 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.318237 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.318285 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.318334 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":50,"threads_started":1}
I20260619 20:52:46.318514 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.314493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39281 (local address 127.0.0.1:41606)
0619 20:52:46.314603 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.314612 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.314649 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.315095 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.315096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.315098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:46.315144 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.315146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.316547 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.316548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.317568 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.317569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.318234 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.318235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.318305 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.318312 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.318396 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.318439 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":28,"thread_start_us":79,"threads_started":1}
I20260619 20:52:46.318706 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.314299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.0.1:37396)
0619 20:52:46.314438 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.314449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.314500 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.315220 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.315220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.315223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:46.315270 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.315271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.316568 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.316569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.317620 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.317623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.318508 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.318509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.318574 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.318579 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.318608 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.318642 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260619 20:52:46.318842 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.314476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.0.98.3:34695)
0619 20:52:46.314835 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.314840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.314851 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:52:46.314891 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.314893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.314908 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.315005 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:46.315192 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.315928 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.317325 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.318050 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.318612 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.318674 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.318699 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.318756 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":82,"threads_started":1}
I20260619 20:52:46.318933 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.314743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.0.98.1:39281)
0619 20:52:46.314927 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.314931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.314940 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:52:46.314978 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.314980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.314995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.315228 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:46.315300 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.316446 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.317665 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.318135 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.318230 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.318290 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.318414 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.318455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260619 20:52:46.319031 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.314639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.98.2:36109)
0619 20:52:46.314798 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.314802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.314810 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:52:46.315109 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.315114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.315133 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.315213 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:46.315331 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.316439 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.317710 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.318412 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.318814 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.318874 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.318900 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.318951 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260619 20:52:46.319689 16066 tablet_service.cc:1467] Processing CreateTablet for tablet c2f262715cd34f06ae7b62fd51816f4e (DEFAULT_TABLE table=test-table [id=d5d48a4987ec46ba84eb21e7a284b95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:46.319685 16200 tablet_service.cc:1467] Processing CreateTablet for tablet c2f262715cd34f06ae7b62fd51816f4e (DEFAULT_TABLE table=test-table [id=d5d48a4987ec46ba84eb21e7a284b95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:46.319967 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f262715cd34f06ae7b62fd51816f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:46.320034 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f262715cd34f06ae7b62fd51816f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:46.322655 16284 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Bootstrap starting.
I20260619 20:52:46.323040 15932 tablet_service.cc:1467] Processing CreateTablet for tablet c2f262715cd34f06ae7b62fd51816f4e (DEFAULT_TABLE table=test-table [id=d5d48a4987ec46ba84eb21e7a284b95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:52:46.323243 16284 tablet_bootstrap.cc:654] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:46.323352 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f262715cd34f06ae7b62fd51816f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:52:46.323797 16284 log.cc:824] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:46.324108 16283 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Bootstrap starting.
I20260619 20:52:46.324534 16284 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: No bootstrap required, opened a new log
I20260619 20:52:46.324604 16284 ts_tablet_manager.cc:1397] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:46.324764 16283 tablet_bootstrap.cc:654] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:46.325357 16283 log.cc:824] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:46.326329 16283 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: No bootstrap required, opened a new log
I20260619 20:52:46.326349 16287 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Bootstrap starting.
I20260619 20:52:46.326411 16283 ts_tablet_manager.cc:1397] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:52:46.326948 16287 tablet_bootstrap.cc:654] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Neither blocks nor log segments found. Creating new log.
I20260619 20:52:46.326822 16284 raft_consensus.cc:348] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.326997 16284 raft_consensus.cc:374] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:46.327035 16284 raft_consensus.cc:729] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f5de920cdb4872b35dad447402a3dc, State: Initialized, Role: FOLLOWER
I20260619 20:52:46.327234 16284 consensus_queue.cc:260] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.327497 16287 log.cc:824] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:52:46.327538 16284 ts_tablet_manager.cc:1428] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:52:46.327668 16266 heartbeater.cc:503] Master 127.0.98.62:34465 was elected leader, sending a full tablet report...
I20260619 20:52:46.328238 16287 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: No bootstrap required, opened a new log
I20260619 20:52:46.328331 16287 ts_tablet_manager.cc:1397] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:52:46.328229 16283 raft_consensus.cc:348] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.328395 16283 raft_consensus.cc:374] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:46.328437 16283 raft_consensus.cc:729] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ff1991f1c4d5891c8ac77c544ca61, State: Initialized, Role: FOLLOWER
I20260619 20:52:46.328545 16283 consensus_queue.cc:260] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.328816 16283 ts_tablet_manager.cc:1428] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:52:46.328859 16132 heartbeater.cc:503] Master 127.0.98.62:34465 was elected leader, sending a full tablet report...
I20260619 20:52:46.330420 16287 raft_consensus.cc:348] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.330596 16287 raft_consensus.cc:374] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:52:46.330641 16287 raft_consensus.cc:729] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c5f912fc6477b985ea848568fb7cc, State: Initialized, Role: FOLLOWER
I20260619 20:52:46.330749 16287 consensus_queue.cc:260] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.331037 15998 heartbeater.cc:503] Master 127.0.98.62:34465 was elected leader, sending a full tablet report...
I20260619 20:52:46.331048 16287 ts_tablet_manager.cc:1428] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 20:52:46.374030 16133 tablet.cc:2369] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:52:46.382373 16291 raft_consensus.cc:482] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:52:46.382545 16291 raft_consensus.cc:504] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.382937 16291 leader_election.cc:288] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
I20260619 20:52:46.386896 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.383129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.98.1:33751)
0619 20:52:46.383189 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.383203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.383263 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.383452 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.383453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.383455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:46.383545 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.383547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.384527 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.384528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.385837 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.385841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.386621 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.386622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.386723 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.386727 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.386783 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.386827 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:52:46.386896 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.383235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33751 (local address 127.0.98.2:36109)
0619 20:52:46.383302 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.383304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.383315 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:52:46.383347 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.383349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.383362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.383428 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:46.383636 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.384411 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.385931 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.386502 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.386575 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.386585 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.386787 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.386827 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:52:46.386958 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.383243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56779 (local address 127.0.98.3:34695)
0619 20:52:46.383291 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.383293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.383584 (+ 291us) server_negotiation.cc:407] Connection header received
0619 20:52:46.383635 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.383636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.383650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.383703 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:52:46.383854 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.384526 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.385931 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.386507 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.386575 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.386585 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.386828 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.386882 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:52:46.387224 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.383129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34695 (local address 127.0.98.1:56779)
0619 20:52:46.383472 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.383483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:52:46.383545 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:52:46.383712 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:52:46.383713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:52:46.383714 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:52:46.383768 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.383770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.384629 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.384630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.385838 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:52:46.385841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.386608 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.386609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:52:46.386690 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.386698 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:52:46.387064 (+ 366us) client_negotiation.cc:240] Negotiation successful
0619 20:52:46.387124 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":148,"thread_start_us":117,"threads_started":1}
I20260619 20:52:46.387282 16086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f262715cd34f06ae7b62fd51816f4e" candidate_uuid: "559c5f912fc6477b985ea848568fb7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ff1991f1c4d5891c8ac77c544ca61" is_pre_election: true
I20260619 20:52:46.387466 16086 raft_consensus.cc:2454] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 559c5f912fc6477b985ea848568fb7cc in term 0.
I20260619 20:52:46.387643 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f262715cd34f06ae7b62fd51816f4e" candidate_uuid: "559c5f912fc6477b985ea848568fb7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f5de920cdb4872b35dad447402a3dc" is_pre_election: true
I20260619 20:52:46.387748 15886 leader_election.cc:302] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ff1991f1c4d5891c8ac77c544ca61, 559c5f912fc6477b985ea848568fb7cc; no voters:
I20260619 20:52:46.387805 16220 raft_consensus.cc:2454] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 559c5f912fc6477b985ea848568fb7cc in term 0.
I20260619 20:52:46.387863 16291 raft_consensus.cc:2781] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:52:46.387935 16291 raft_consensus.cc:482] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:52:46.387974 16291 raft_consensus.cc:3037] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:46.388790 16291 raft_consensus.cc:504] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.388988 16291 leader_election.cc:288] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 1 election: Requested vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
I20260619 20:52:46.389099 16086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f262715cd34f06ae7b62fd51816f4e" candidate_uuid: "559c5f912fc6477b985ea848568fb7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ff1991f1c4d5891c8ac77c544ca61"
I20260619 20:52:46.389099 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f262715cd34f06ae7b62fd51816f4e" candidate_uuid: "559c5f912fc6477b985ea848568fb7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f5de920cdb4872b35dad447402a3dc"
I20260619 20:52:46.389192 16086 raft_consensus.cc:3037] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:46.389189 16220 raft_consensus.cc:3037] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:52:46.389979 16086 raft_consensus.cc:2454] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 559c5f912fc6477b985ea848568fb7cc in term 1.
I20260619 20:52:46.390030 16220 raft_consensus.cc:2454] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 559c5f912fc6477b985ea848568fb7cc in term 1.
I20260619 20:52:46.390220 15886 leader_election.cc:302] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ff1991f1c4d5891c8ac77c544ca61, 559c5f912fc6477b985ea848568fb7cc; no voters:
I20260619 20:52:46.390329 16291 raft_consensus.cc:2781] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:52:46.390491 16291 raft_consensus.cc:686] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 LEADER]: Becoming Leader. State: Replica: 559c5f912fc6477b985ea848568fb7cc, State: Running, Role: LEADER
I20260619 20:52:46.390563 16291 consensus_queue.cc:237] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:52:46.391392 15807 catalog_manager.cc:5495] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 559c5f912fc6477b985ea848568fb7cc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "559c5f912fc6477b985ea848568fb7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } health_report { overall_health: HEALTHY } } }
I20260619 20:52:46.409799 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.407445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.98.1:39281)
0619 20:52:46.407486 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:52:46.407488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:52:46.407496 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:52:46.407561 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:52:46.407562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:52:46.407580 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:52:46.407641 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:52:46.407737 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.408234 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.409091 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:52:46.409269 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:52:46.409360 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:52:46.409514 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:52:46.409609 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:52:46.409674 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:52:46.409725 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:52:46.409743 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:52:46.411476 16220 raft_consensus.cc:1261] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Refusing update from remote peer 559c5f912fc6477b985ea848568fb7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:46.411830 16086 raft_consensus.cc:1261] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Refusing update from remote peer 559c5f912fc6477b985ea848568fb7cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:52:46.411839 16293 consensus_queue.cc:1035] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:46.412111 16293 consensus_queue.cc:1035] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:52:46.414310 16296 mvcc.cc:205] Tried to move back new op lower bound from 7298672092818960384 to 7298672092736516096. Current Snapshot: MvccSnapshot[applied={T|T < 7298672092818960384}]
I20260619 20:52:46.414525 16297 mvcc.cc:205] Tried to move back new op lower bound from 7298672092818960384 to 7298672092736516096. Current Snapshot: MvccSnapshot[applied={T|T < 7298672092818960384}]
I20260619 20:52:46.414595 16298 mvcc.cc:205] Tried to move back new op lower bound from 7298672092818960384 to 7298672092736516096. Current Snapshot: MvccSnapshot[applied={T|T < 7298672092818960384}]
I20260619 20:52:46.416944 15807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:46.417030 15807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:52:46.417794 15807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Sending DeleteTablet for 3 replicas of tablet c2f262715cd34f06ae7b62fd51816f4e
I20260619 20:52:46.418133 16066 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f262715cd34f06ae7b62fd51816f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37396
I20260619 20:52:46.418133 15932 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f262715cd34f06ae7b62fd51816f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41606
I20260619 20:52:46.418139 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f262715cd34f06ae7b62fd51816f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260619 20:52:46.418370 16306 tablet_replica.cc:330] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: stopping tablet replica
I20260619 20:52:46.418386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 15869
I20260619 20:52:46.418463 16306 raft_consensus.cc:2229] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:52:46.418563 16306 raft_consensus.cc:2258] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:46.418691 16307 tablet_replica.cc:330] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: stopping tablet replica
I20260619 20:52:46.418758 16307 raft_consensus.cc:2229] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:46.418915 16307 raft_consensus.cc:2258] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:46.419309 16307 ts_tablet_manager.cc:1905] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:52:46.420753 16307 ts_tablet_manager.cc:1918] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:52:46.420814 16307 log.cc:1196] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/c2f262715cd34f06ae7b62fd51816f4e
I20260619 20:52:46.421098 16307 ts_tablet_manager.cc:1939] T c2f262715cd34f06ae7b62fd51816f4e P 89f5de920cdb4872b35dad447402a3dc: Deleting consensus metadata
I20260619 20:52:46.421536 15794 catalog_manager.cc:4841] TS 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): tablet c2f262715cd34f06ae7b62fd51816f4e (table test-table [id=d5d48a4987ec46ba84eb21e7a284b95f]) successfully deleted
W20260619 20:52:46.424249 15791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39281 (error 108)
I20260619 20:52:46.424369 16308 tablet_replica.cc:330] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: stopping tablet replica
I20260619 20:52:46.424482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16003
I20260619 20:52:46.424490 16308 raft_consensus.cc:2229] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:52:46.424600 16308 raft_consensus.cc:2258] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:52:46.425885 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.425744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39281 (local address 127.0.0.1:41626)
0619 20:52:46.425808 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.425847 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 20:52:46.426054 15791 catalog_manager.cc:4584] TS 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2f262715cd34f06ae7b62fd51816f4e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39281: connect: Connection refused (error 111)
I20260619 20:52:46.429080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16137
I20260619 20:52:46.429134 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:52:46.428966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.0.1:37402)
0619 20:52:46.429040 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:52:46.429100 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 20:52:46.433553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 15775
I20260619 20:53:06.439270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34465
--webserver_interface=127.0.98.62
--webserver_port=43349
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:06.519411 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:06.519613 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:06.519634 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:06.520757 16310 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:53:06.520799 16310 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:53:06.520812 16310 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:53:06.520824 16310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:06.520838 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:06.520891 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:06.520911 16310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:06.520936 16310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:06.522437 16310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:06.522683 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:06.522913 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:06.531870 16317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.531951 16316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:06.532125 16310 server_base.cc:1034] running on GCE node
W20260619 20:53:06.531893 16319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:06.532699 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:06.532999 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:06.534162 16310 hybrid_clock.cc:648] HybridClock initialized: now 1781902386534147 us; error 28 us; skew 500 ppm
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:06.539934 16310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:53:06.540364 16310 webserver.cc:460] Webserver started at http://127.0.98.62:43349/ using document root <none> and password file <none>
I20260619 20:53:06.540596 16310 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:06.540673 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:06.542089 16310 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:06.542743 16326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.542896 16310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:06.542999 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "48f4db8e2e4c46c693e294784fc1a115"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "c31c809ade349cc2f6d0b7cc05558908"
server_key_iv: "0e11a92e77cb0694f08233997d94ab83"
server_key_version: "encryptionkey@0"
I20260619 20:53:06.543394 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:06.574766 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:06.576713 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:06.576877 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:06.580821 16310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34465
I20260619 20:53:06.580863 16378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34465 every 8 connection(s)
I20260619 20:53:06.581264 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:06.582723 16379 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:53:06.583369 16379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap starting.
I20260619 20:53:06.584285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16310
I20260619 20:53:06.584591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39281
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34465
--webserver_port=35207
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:53:06.592360 16379 log.cc:824] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:06.594435 16379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:53:06.594719 16379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap complete.
I20260619 20:53:06.596956 16379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:06.597180 16379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f4db8e2e4c46c693e294784fc1a115, State: Initialized, Role: FOLLOWER
I20260619 20:53:06.597373 16379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:06.597451 16379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:06.597517 16379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:06.597568 16379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:53:06.598603 16379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:06.598687 16379 leader_election.cc:302] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f4db8e2e4c46c693e294784fc1a115; no voters:
I20260619 20:53:06.598903 16379 leader_election.cc:288] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:53:06.598937 16383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:53:06.599104 16383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 LEADER]: Becoming Leader. State: Replica: 48f4db8e2e4c46c693e294784fc1a115, State: Running, Role: LEADER
I20260619 20:53:06.599164 16379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:06.599238 16383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:06.599506 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f4db8e2e4c46c693e294784fc1a115. Latest consensus state: current_term: 2 leader_uuid: "48f4db8e2e4c46c693e294784fc1a115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } } }
I20260619 20:53:06.599592 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:06.599510 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48f4db8e2e4c46c693e294784fc1a115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } } }
I20260619 20:53:06.599802 16389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:06.599824 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:06.600596 16389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:06.600705 16389 catalog_manager.cc:1259] Loaded cluster ID: 5a2389d00e9b448ebf411e3f01aac3c5
I20260619 20:53:06.600742 16389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:06.601915 16389 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:06.602097 16389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Loaded TSK: 0
I20260619 20:53:06.620285 16389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Generated new TSK 1
I20260619 20:53:06.620388 16389 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:53:06.667526 16381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:06.667743 16381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:06.667784 16381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:06.668929 16381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:06.668991 16381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:06.669039 16381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:06.670493 16381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35207
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:06.670739 16381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:06.670964 16381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:06.676646 16406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.676658 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.676721 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:06.677026 16381 server_base.cc:1034] running on GCE node
I20260619 20:53:06.677213 16381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:06.677456 16381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:06.678617 16381 hybrid_clock.cc:648] HybridClock initialized: now 1781902386678602 us; error 33 us; skew 500 ppm
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:06.681223 16381 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:53:06.681752 16381 webserver.cc:460] Webserver started at http://127.0.98.1:35207/ using document root <none> and password file <none>
I20260619 20:53:06.682003 16381 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:06.682093 16381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:06.683849 16381 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:53:06.684609 16416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.684787 16381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:06.684882 16381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "559c5f912fc6477b985ea848568fb7cc"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "e8c3de60401653cd786aada9e0473712"
server_key_iv: "f2fe58eb80f444292d2f78543d8eddc3"
server_key_version: "encryptionkey@0"
I20260619 20:53:06.685226 16381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:06.694449 16381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:06.696694 16381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:06.696890 16381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:06.697134 16381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:06.697649 16423 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:53:06.699509 16381 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:53:06.699580 16381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:53:06.699712 16381 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:53:06.700388 16381 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:53:06.700444 16381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:53:06.700511 16423 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Bootstrap starting.
I20260619 20:53:06.707705 16381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39281
I20260619 20:53:06.708066 16381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:06.709753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16381
I20260619 20:53:06.710037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36109
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34465
--webserver_port=46411
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:53:06.711521 16423 log.cc:824] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:06.713435 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39281 every 8 connection(s)
I20260619 20:53:06.714299 16423 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:53:06.714603 16423 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Bootstrap complete.
I20260619 20:53:06.715787 16423 ts_tablet_manager.cc:1397] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260619 20:53:06.717116 16423 raft_consensus.cc:348] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:06.717453 16423 raft_consensus.cc:729] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c5f912fc6477b985ea848568fb7cc, State: Initialized, Role: FOLLOWER
I20260619 20:53:06.717614 16423 consensus_queue.cc:260] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:06.717869 16423 ts_tablet_manager.cc:1428] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:06.725703 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.708864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43251 (local address 127.0.98.62:34465)
0619 20:53:06.713962 (+ 5098us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.713971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.714012 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:53:06.714593 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.714599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.714642 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.714801 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:06.715447 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.716406 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.717435 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.718318 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.718431 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.721459 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:06.721463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:06.721463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:06.721476 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:06.723548 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.724106 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.724109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.724109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.724219 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.724532 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.724533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.724534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.724824 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:06.724908 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.725336 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.725476 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5046,"thread_start_us":71,"threads_started":1}
I20260619 20:53:06.725708 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.708793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:43251)
0619 20:53:06.713689 (+ 4896us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:06.713720 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:06.714279 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:06.714830 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:06.714834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:06.715134 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:06.715325 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.715330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.716511 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.716512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.717278 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.717280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.718420 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.718421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.718664 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.719372 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:06.719390 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:06.721347 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:06.723640 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.723649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.723651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.723997 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.724293 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.724294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.724294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.724445 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.724905 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:06.724913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:06.725103 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 20:53:06.725222 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:53:06.725436 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4851,"thread_start_us":115,"threads_started":1}
I20260619 20:53:06.727089 16531 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34465
I20260619 20:53:06.727187 16531 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:06.727360 16531 heartbeater.cc:511] Master 127.0.98.62:34465 requested a full tablet report, sending...
I20260619 20:53:06.728289 16343 ts_manager.cc:194] Registered new tserver with Master: 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281)
I20260619 20:53:06.729475 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43251
I20260619 20:53:06.730433 16531 heartbeater.cc:503] Master 127.0.98.62:34465 was elected leader, sending a full tablet report...
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 20:53:06.739720 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.728897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39281 (local address 127.0.0.1:41968)
0619 20:53:06.729640 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:06.729677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:06.729746 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:06.730146 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:06.730152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:06.730460 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:06.730651 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.730652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.731950 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.731951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.732932 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.732935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.733560 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.733563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.733652 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.734764 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:06.734770 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:06.736443 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:06.738561 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.738567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.738569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.738807 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.739040 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.739041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.739041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.739108 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.739384 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:06.739390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:06.739440 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:53:06.739522 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:53:06.739629 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":113,"threads_started":1}
I20260619 20:53:06.739744 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.729381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.0.98.1:39281)
0619 20:53:06.729961 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.729965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.729981 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:53:06.730010 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.730012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.730025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.730088 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:06.730723 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.731858 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.733022 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.733490 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.733561 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.736542 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:06.736552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:06.736552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:06.736561 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:06.738463 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.738898 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.738900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.738901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.738958 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.739167 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.739168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.739168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.739322 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:06.739392 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.739562 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.739650 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":73,"threads_started":1}
I20260619 20:53:06.740445 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f262715cd34f06ae7b62fd51816f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41968
I20260619 20:53:06.741050 16540 tablet_replica.cc:330] stopping tablet replica
I20260619 20:53:06.741147 16540 raft_consensus.cc:2229] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:06.741210 16540 raft_consensus.cc:2258] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:06.741523 16540 ts_tablet_manager.cc:1905] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:06.743777 16540 ts_tablet_manager.cc:1918] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:06.743849 16540 log.cc:1196] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal/wals/c2f262715cd34f06ae7b62fd51816f4e
I20260619 20:53:06.744061 16540 ts_tablet_manager.cc:1939] T c2f262715cd34f06ae7b62fd51816f4e P 559c5f912fc6477b985ea848568fb7cc: Deleting consensus metadata
I20260619 20:53:06.744685 16327 catalog_manager.cc:4841] TS 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281): tablet c2f262715cd34f06ae7b62fd51816f4e (table test-table [id=d5d48a4987ec46ba84eb21e7a284b95f]) successfully deleted
W20260619 20:53:06.801535 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:06.801745 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:06.801785 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:06.803057 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:06.803117 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:06.803161 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:06.804582 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46411
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:06.804822 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:06.805043 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:53:06.809849 16535 server_base.cc:1034] running on GCE node
W20260619 20:53:06.809813 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.809831 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.809813 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:06.810155 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:06.810369 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:06.811501 16535 hybrid_clock.cc:648] HybridClock initialized: now 1781902386811494 us; error 27 us; skew 500 ppm
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:06.813866 16535 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:53:06.814268 16535 webserver.cc:460] Webserver started at http://127.0.98.2:46411/ using document root <none> and password file <none>
I20260619 20:53:06.814465 16535 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:06.814538 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:06.815809 16535 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:06.816419 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.816596 16535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.816676 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "445ff1991f1c4d5891c8ac77c544ca61"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "77f291f52b2bf1ee55c8673aa19575b5"
server_key_iv: "8547f27e137d42150e927346f0933ccb"
server_key_version: "encryptionkey@0"
I20260619 20:53:06.816989 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:06.842633 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:06.844485 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:06.844590 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:06.844769 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:06.845171 16562 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:53:06.846730 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:53:06.846787 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:53:06.846832 16535 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:53:06.847584 16535 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:53:06.847633 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:53:06.847703 16562 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Bootstrap starting.
I20260619 20:53:06.853943 16535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36109
I20260619 20:53:06.854019 16669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36109 every 8 connection(s)
I20260619 20:53:06.854319 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:06.858455 16562 log.cc:824] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:06.860894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16535
I20260619 20:53:06.860926 16562 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:53:06.861186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34695
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34465
--webserver_port=40197
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:06.861235 16562 tablet_bootstrap.cc:492] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Bootstrap complete.
I20260619 20:53:06.862349 16562 ts_tablet_manager.cc:1397] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260619 20:53:06.863297 16562 raft_consensus.cc:348] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:06.863534 16562 raft_consensus.cc:729] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ff1991f1c4d5891c8ac77c544ca61, State: Initialized, Role: FOLLOWER
I20260619 20:53:06.863672 16562 consensus_queue.cc:260] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:06.863947 16562 ts_tablet_manager.cc:1428] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 20:53:06.865041 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.855063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57067 (local address 127.0.98.62:34465)
0619 20:53:06.855109 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.855111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.855352 (+ 241us) server_negotiation.cc:407] Connection header received
0619 20:53:06.856048 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.856049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.856061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.856104 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:06.857001 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.857538 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.859017 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.859229 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.859282 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.861783 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:06.861784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:06.861784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:06.861792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:06.863544 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.864061 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.864063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.864063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.864138 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.864393 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.864394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.864395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.864517 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:06.864588 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.864846 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.864951 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:53:06.865177 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.855015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:57067)
0619 20:53:06.855193 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:06.855246 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:06.855936 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:06.856173 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:06.856178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:06.856582 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:06.856908 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.856916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.857652 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.857653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.858936 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.858939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.859331 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.859332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.859485 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.859998 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:06.860007 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:06.861686 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:06.863642 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.863648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.863650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.863964 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.864201 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.864201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.864202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.864310 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.864581 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:06.864587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:06.864720 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:53:06.864825 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:53:06.864976 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260619 20:53:06.865675 16670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34465
I20260619 20:53:06.865778 16670 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:06.865960 16670 heartbeater.cc:511] Master 127.0.98.62:34465 requested a full tablet report, sending...
I20260619 20:53:06.866554 16343 ts_manager.cc:194] Registered new tserver with Master: 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
I20260619 20:53:06.867156 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57067
I20260619 20:53:06.868254 16670 heartbeater.cc:503] Master 127.0.98.62:34465 was elected leader, sending a full tablet report...
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 20:53:06.876972 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.866910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.0.98.2:36109)
0619 20:53:06.867290 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.867294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.867303 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:06.867368 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.867370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.867384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.867840 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:06.868402 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.869682 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.870875 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.871344 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.871420 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.873707 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:06.873714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:06.873714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:06.873720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:06.875777 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.876284 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.876287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.876287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.876346 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.876543 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.876543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.876544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.876665 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:06.876722 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.876853 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.876915 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":82,"threads_started":1}
I20260619 20:53:06.877019 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.866842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.0.1:54404)
0619 20:53:06.866916 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:06.866927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:06.866968 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:06.867558 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:06.867559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:06.867716 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:06.867760 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.867762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.869763 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.869764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.870763 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.870766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.871432 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.871435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.871539 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.871975 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:06.871980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:06.873610 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:06.875899 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.875900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.875901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.876116 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.876426 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.876427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.876427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.876490 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.876719 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:06.876720 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:06.876761 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:53:06.876831 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:53:06.876926 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 20:53:06.877406 16604 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f262715cd34f06ae7b62fd51816f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:52:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54404
I20260619 20:53:06.877943 16677 tablet_replica.cc:330] stopping tablet replica
I20260619 20:53:06.878041 16677 raft_consensus.cc:2229] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:06.878104 16677 raft_consensus.cc:2258] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:06.878371 16677 ts_tablet_manager.cc:1905] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:06.880324 16677 ts_tablet_manager.cc:1918] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:06.880386 16677 log.cc:1196] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/c2f262715cd34f06ae7b62fd51816f4e
I20260619 20:53:06.880635 16677 ts_tablet_manager.cc:1939] T c2f262715cd34f06ae7b62fd51816f4e P 445ff1991f1c4d5891c8ac77c544ca61: Deleting consensus metadata
I20260619 20:53:06.881129 16328 catalog_manager.cc:4841] TS 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): tablet c2f262715cd34f06ae7b62fd51816f4e (table test-table [id=d5d48a4987ec46ba84eb21e7a284b95f]) successfully deleted
W20260619 20:53:06.946659 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:06.946857 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:06.946879 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:06.947921 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:06.947963 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:06.947989 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:06.949335 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40197
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:06.949504 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:06.949682 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:06.954360 16685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.954378 16682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:06.954380 16683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:06.954646 16674 server_base.cc:1034] running on GCE node
I20260619 20:53:06.954836 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:06.955116 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:06.956267 16674 hybrid_clock.cc:648] HybridClock initialized: now 1781902386956257 us; error 30 us; skew 500 ppm
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:06.958699 16674 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:53:06.959106 16674 webserver.cc:460] Webserver started at http://127.0.98.3:40197/ using document root <none> and password file <none>
I20260619 20:53:06.959333 16674 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:06.959398 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:06.960703 16674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:06.961253 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.961460 16674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:06.961541 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "89f5de920cdb4872b35dad447402a3dc"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "f02c19d73151fee0b2e3a1aa7cdd8389"
server_key_iv: "dc996e7a70751b27fc5010cfd2b0761d"
server_key_version: "encryptionkey@0"
I20260619 20:53:06.961826 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:06.972043 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:06.973788 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:06.973891 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:06.974061 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:06.974287 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:06.974316 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.974336 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:06.974349 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:06.979974 16674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34695
I20260619 20:53:06.980041 16805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34695 every 8 connection(s)
I20260619 20:53:06.980350 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:53:06 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:06.986722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16674
I20260619 20:53:06.989692 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.981050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54173 (local address 127.0.98.62:34465)
0619 20:53:06.981110 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.981111 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.981310 (+ 199us) server_negotiation.cc:407] Connection header received
0619 20:53:06.981825 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.981826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.981838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.981889 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:06.982782 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.983281 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.984247 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.984461 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.984544 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.986540 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:06.986541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:06.986542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:06.986549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:06.988306 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.988712 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.988713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.988714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.988773 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.989005 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.989006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.989007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.989119 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:06.989191 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.989513 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.989602 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:53:06.989917 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.981035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:54173)
0619 20:53:06.981192 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:06.981224 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:06.981720 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:06.981964 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:06.981968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:06.982508 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:06.982687 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.982692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.983370 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.983371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.984159 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:06.984161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.984551 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.984552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:06.984668 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.985155 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:06.985163 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:06.986451 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:06.988383 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.988389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.988392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.988630 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.988837 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:06.988838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:06.988838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:06.988944 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:06.989181 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:06.989187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:06.989360 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 20:53:06.989473 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 20:53:06.989653 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260619 20:53:06.990375 16806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34465
I20260619 20:53:06.990473 16806 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:06.990661 16806 heartbeater.cc:511] Master 127.0.98.62:34465 requested a full tablet report, sending...
I20260619 20:53:06.991043 16343 ts_manager.cc:194] Registered new tserver with Master: 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695)
I20260619 20:53:06.991400 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54173
I20260619 20:53:06.993732 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.987167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.98.62:34465)
0619 20:53:06.987286 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.987288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.987420 (+ 132us) server_negotiation.cc:407] Connection header received
0619 20:53:06.987488 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.987489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.987501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.987569 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:06.988363 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.988830 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.989661 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.989932 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.990020 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.991430 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:06.991431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:06.991431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:06.991438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:06.992758 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.993103 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.993104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.993104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.993150 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:06.993328 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:06.993329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:06.993329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:06.993426 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:06.993559 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.993620 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.993675 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260619 20:53:06.994447 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:06.996774 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:06.994770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.0.98.62:34465)
0619 20:53:06.994818 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:06.994820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:06.994904 (+ 84us) server_negotiation.cc:407] Connection header received
0619 20:53:06.994953 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:06.994953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:06.994977 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:06.995047 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:06.995113 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.995537 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.996185 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:06.996377 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:06.996423 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:06.996543 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:06.996622 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:06.996691 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:06.996719 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 20:53:06.996736 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:53:06.997046 16343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:06.998574 16343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:07.000427 16450 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffa3da600834c3383b68a264c15e030 (DEFAULT_TABLE table=test-table [id=7e5c50cf831341b2b7937dd51de16dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:07.000495 16604 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffa3da600834c3383b68a264c15e030 (DEFAULT_TABLE table=test-table [id=7e5c50cf831341b2b7937dd51de16dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:07.000620 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffa3da600834c3383b68a264c15e030. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:07.000674 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffa3da600834c3383b68a264c15e030. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:07.001938 16562 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Bootstrap starting.
I20260619 20:53:07.002182 16423 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Bootstrap starting.
I20260619 20:53:07.002310 16562 tablet_bootstrap.cc:654] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:07.002460 16423 tablet_bootstrap.cc:654] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:07.002997 16423 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: No bootstrap required, opened a new log
I20260619 20:53:07.003048 16423 ts_tablet_manager.cc:1397] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:53:07.003059 16562 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: No bootstrap required, opened a new log
I20260619 20:53:07.003101 16562 ts_tablet_manager.cc:1397] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:53:07.003358 16423 raft_consensus.cc:348] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.003433 16423 raft_consensus.cc:374] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:07.003474 16423 raft_consensus.cc:729] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c5f912fc6477b985ea848568fb7cc, State: Initialized, Role: FOLLOWER
I20260619 20:53:07.003516 16423 consensus_queue.cc:260] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.003500 16562 raft_consensus.cc:348] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.003572 16562 raft_consensus.cc:374] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:07.003621 16562 raft_consensus.cc:729] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ff1991f1c4d5891c8ac77c544ca61, State: Initialized, Role: FOLLOWER
I20260619 20:53:07.003621 16423 ts_tablet_manager.cc:1428] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:07.003665 16562 consensus_queue.cc:260] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.003748 16562 ts_tablet_manager.cc:1428] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:07.004165 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.000175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34695 (local address 127.0.0.1:48522)
0619 20:53:07.000227 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:07.000237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:07.000269 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:07.000590 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:07.000591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:07.000592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:07.000629 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:07.000631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.001802 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.001804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:07.002921 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:07.002923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.003932 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.003933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:07.003994 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.004012 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 20:53:07.004063 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:53:07.004107 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:53:07.004457 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.000279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.0.98.3:34695)
0619 20:53:07.000435 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:07.000440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:07.000450 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:53:07.000482 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:07.000484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:07.000498 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:07.000582 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:07.000698 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.001712 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.003027 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.003844 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.003934 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.003991 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:07.004335 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 20:53:07.004374 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260619 20:53:07.005110 16740 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffa3da600834c3383b68a264c15e030 (DEFAULT_TABLE table=test-table [id=7e5c50cf831341b2b7937dd51de16dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:07.005434 16740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffa3da600834c3383b68a264c15e030. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:07.007943 16815 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Bootstrap starting.
I20260619 20:53:07.008478 16815 tablet_bootstrap.cc:654] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:07.008962 16815 log.cc:824] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:07.009644 16815 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: No bootstrap required, opened a new log
I20260619 20:53:07.009721 16815 ts_tablet_manager.cc:1397] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:07.011168 16815 raft_consensus.cc:348] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.011313 16815 raft_consensus.cc:374] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:07.011358 16815 raft_consensus.cc:729] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f5de920cdb4872b35dad447402a3dc, State: Initialized, Role: FOLLOWER
I20260619 20:53:07.011471 16815 consensus_queue.cc:260] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.011729 16815 ts_tablet_manager.cc:1428] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:07.011761 16806 heartbeater.cc:503] Master 127.0.98.62:34465 was elected leader, sending a full tablet report...
I20260619 20:53:07.037225 16675 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:07.037405 16675 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.037719 16675 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281)
I20260619 20:53:07.040525 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.037871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34695 (local address 127.0.98.2:60205)
0619 20:53:07.037927 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:07.037938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:07.037997 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:07.038186 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:07.038188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:07.038189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:07.038264 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:07.038266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.038913 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.038914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:07.039711 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:07.039713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.040271 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.040272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:07.040342 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.040366 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 20:53:07.040418 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:53:07.040453 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:53:07.040535 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.037948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60205 (local address 127.0.98.3:34695)
0619 20:53:07.037990 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:07.037992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:07.038003 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:53:07.038076 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:07.038077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:07.038088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:07.038154 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:07.038334 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.038837 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.039789 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.040198 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.040249 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.040257 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:07.040431 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 20:53:07.040470 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:53:07.040694 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.037953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52143 (local address 127.0.98.1:39281)
0619 20:53:07.038007 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:07.038009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:07.038143 (+ 134us) server_negotiation.cc:407] Connection header received
0619 20:53:07.038213 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:07.038215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:07.038241 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:07.038306 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:07.038470 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.038967 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.039897 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.040355 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.040426 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.040500 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:07.040583 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:53:07.040626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:53:07.040748 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.037871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39281 (local address 127.0.98.2:52143)
0619 20:53:07.038080 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:07.038091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:07.038143 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:07.038331 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:07.038332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:07.038334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:07.038406 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:07.038407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.039043 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.039044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:07.039807 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:07.039809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.040454 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.040455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:07.040519 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.040522 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:07.040578 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:53:07.040620 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":19,"thread_start_us":111,"threads_started":1}
I20260619 20:53:07.040856 16760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa3da600834c3383b68a264c15e030" candidate_uuid: "445ff1991f1c4d5891c8ac77c544ca61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f5de920cdb4872b35dad447402a3dc" is_pre_election: true
I20260619 20:53:07.041030 16760 raft_consensus.cc:2454] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ff1991f1c4d5891c8ac77c544ca61 in term 0.
I20260619 20:53:07.041065 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa3da600834c3383b68a264c15e030" candidate_uuid: "445ff1991f1c4d5891c8ac77c544ca61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c5f912fc6477b985ea848568fb7cc" is_pre_election: true
I20260619 20:53:07.041220 16485 raft_consensus.cc:2454] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ff1991f1c4d5891c8ac77c544ca61 in term 0.
I20260619 20:53:07.041265 16559 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ff1991f1c4d5891c8ac77c544ca61, 89f5de920cdb4872b35dad447402a3dc; no voters:
I20260619 20:53:07.041399 16675 raft_consensus.cc:2781] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:07.041445 16675 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:07.041474 16675 raft_consensus.cc:3037] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:07.042161 16675 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.042300 16675 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [CANDIDATE]: Term 1 election: Requested vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281)
I20260619 20:53:07.042446 16760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa3da600834c3383b68a264c15e030" candidate_uuid: "445ff1991f1c4d5891c8ac77c544ca61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f5de920cdb4872b35dad447402a3dc"
I20260619 20:53:07.042433 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffa3da600834c3383b68a264c15e030" candidate_uuid: "445ff1991f1c4d5891c8ac77c544ca61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559c5f912fc6477b985ea848568fb7cc"
I20260619 20:53:07.042514 16485 raft_consensus.cc:3037] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:07.042515 16760 raft_consensus.cc:3037] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:07.043365 16485 raft_consensus.cc:2454] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ff1991f1c4d5891c8ac77c544ca61 in term 1.
I20260619 20:53:07.043358 16760 raft_consensus.cc:2454] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ff1991f1c4d5891c8ac77c544ca61 in term 1.
I20260619 20:53:07.043530 16556 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ff1991f1c4d5891c8ac77c544ca61, 559c5f912fc6477b985ea848568fb7cc; no voters:
I20260619 20:53:07.043617 16675 raft_consensus.cc:2781] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:07.043771 16675 raft_consensus.cc:686] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 LEADER]: Becoming Leader. State: Replica: 445ff1991f1c4d5891c8ac77c544ca61, State: Running, Role: LEADER
I20260619 20:53:07.043836 16675 consensus_queue.cc:237] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:07.044570 16343 catalog_manager.cc:5495] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "445ff1991f1c4d5891c8ac77c544ca61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } health_report { overall_health: UNKNOWN } } }
I20260619 20:53:07.051143 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.048868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.98.2:36109)
0619 20:53:07.048911 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:07.048913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:07.048964 (+ 51us) server_negotiation.cc:407] Connection header received
0619 20:53:07.049048 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:07.049049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:07.049065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:07.049139 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:07.049235 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.049738 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.050464 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.050669 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.050734 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.050892 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:07.050943 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 20:53:07.051078 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54416: token expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 20:53:07.051191 16676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54416: token expired
I20260619 20:53:07.056607 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.051609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.98.62:34465)
0619 20:53:07.051649 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:07.051651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:07.051665 (+ 14us) server_negotiation.cc:407] Connection header received
0619 20:53:07.051683 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:07.051683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:07.051691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:07.051745 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:07.052174 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.052632 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.053234 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.053447 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.053496 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.054515 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:07.054516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:07.054517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:07.054522 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:07.055770 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:07.056080 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:07.056081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:07.056081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:07.056128 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:07.056252 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:07.056252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:07.056252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:07.056341 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:07.056411 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:07.056494 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:53:07.056551 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:53:07.059442 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.057475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.0.98.2:36109)
0619 20:53:07.057525 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:07.057527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:07.057535 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:07.057557 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:07.057558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:07.057566 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:07.057616 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:07.057715 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.058121 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.058789 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:07.058978 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:07.059043 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:07.059170 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:07.059250 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:07.059306 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:07.059365 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:53:07.059379 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:53:07.060735 16485 raft_consensus.cc:1261] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Refusing update from remote peer 445ff1991f1c4d5891c8ac77c544ca61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:07.060735 16760 raft_consensus.cc:1261] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Refusing update from remote peer 445ff1991f1c4d5891c8ac77c544ca61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:07.061044 16819 consensus_queue.cc:1035] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:07.061170 16675 consensus_queue.cc:1035] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:07.062968 16822 mvcc.cc:205] Tried to move back new op lower bound from 7298672177398796288 to 7298672177332310016. Current Snapshot: MvccSnapshot[applied={T|T < 7298672177398796288}]
I20260619 20:53:07.063661 16823 mvcc.cc:205] Tried to move back new op lower bound from 7298672177398796288 to 7298672177332310016. Current Snapshot: MvccSnapshot[applied={T|T < 7298672177398796288}]
I20260619 20:53:07.065270 16343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:07.065418 16343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:07.066102 16343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Sending DeleteTablet for 3 replicas of tablet 1ffa3da600834c3383b68a264c15e030
I20260619 20:53:07.066396 16740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffa3da600834c3383b68a264c15e030 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48522
I20260619 20:53:07.066424 16604 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffa3da600834c3383b68a264c15e030 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54404
I20260619 20:53:07.066447 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffa3da600834c3383b68a264c15e030 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41968
I20260619 20:53:07.066522 16677 tablet_replica.cc:330] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: stopping tablet replica
I20260619 20:53:07.066526 16540 tablet_replica.cc:330] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: stopping tablet replica
I20260619 20:53:07.066571 16677 raft_consensus.cc:2229] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:53:07.066581 16540 raft_consensus.cc:2229] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:07.066599 16832 tablet_replica.cc:330] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: stopping tablet replica
I20260619 20:53:07.066635 16540 raft_consensus.cc:2258] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:07.066634 16677 raft_consensus.cc:2258] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:07.066766 16832 raft_consensus.cc:2229] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:07.066888 16832 raft_consensus.cc:2258] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:07.066951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16381
I20260619 20:53:07.066962 16677 ts_tablet_manager.cc:1905] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:07.067937 16832 ts_tablet_manager.cc:1905] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:07.068452 16677 ts_tablet_manager.cc:1918] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:07.068503 16677 log.cc:1196] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/1ffa3da600834c3383b68a264c15e030
I20260619 20:53:07.068710 16677 ts_tablet_manager.cc:1939] T 1ffa3da600834c3383b68a264c15e030 P 445ff1991f1c4d5891c8ac77c544ca61: Deleting consensus metadata
I20260619 20:53:07.069063 16328 catalog_manager.cc:4841] TS 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): tablet 1ffa3da600834c3383b68a264c15e030 (table test-table [id=7e5c50cf831341b2b7937dd51de16dac]) successfully deleted
I20260619 20:53:07.069969 16832 ts_tablet_manager.cc:1918] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:07.070043 16832 log.cc:1196] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/1ffa3da600834c3383b68a264c15e030
I20260619 20:53:07.070302 16832 ts_tablet_manager.cc:1939] T 1ffa3da600834c3383b68a264c15e030 P 89f5de920cdb4872b35dad447402a3dc: Deleting consensus metadata
I20260619 20:53:07.070663 16330 catalog_manager.cc:4841] TS 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): tablet 1ffa3da600834c3383b68a264c15e030 (table test-table [id=7e5c50cf831341b2b7937dd51de16dac]) successfully deleted
W20260619 20:53:07.074146 16327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39281 (error 108)
I20260619 20:53:07.074609 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:07.074491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39281 (local address 127.0.0.1:41976)
0619 20:53:07.074540 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:07.074575 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:53:07.074633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16535
W20260619 20:53:07.077677 16327 catalog_manager.cc:4584] TS 559c5f912fc6477b985ea848568fb7cc (127.0.98.1:39281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ffa3da600834c3383b68a264c15e030: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39281: connect: Connection refused (error 111)
I20260619 20:53:07.079484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16674
I20260619 20:53:07.084412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16310
I20260619 20:53:27.089443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34465
--webserver_interface=127.0.98.62
--webserver_port=43349
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:27.169234 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:27.169510 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:27.169536 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:27.170804 16834 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:53:27.170842 16834 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:53:27.170855 16834 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:53:27.170867 16834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:27.170917 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:27.170946 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:27.170969 16834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:27.170994 16834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:27.172456 16834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:27.172704 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:27.172943 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:27.177733 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:27.177752 16839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.177865 16834 server_base.cc:1034] running on GCE node
W20260619 20:53:27.177747 16840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.178176 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:27.178453 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:27.179613 16834 hybrid_clock.cc:648] HybridClock initialized: now 1781902407179594 us; error 50 us; skew 500 ppm
Jun 19 20:53:27 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:27.182066 16834 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:53:27.182451 16834 webserver.cc:460] Webserver started at http://127.0.98.62:43349/ using document root <none> and password file <none>
I20260619 20:53:27.182631 16834 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:27.182672 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:27.183907 16834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:27.184504 16849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.184684 16834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:53:27.184753 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "48f4db8e2e4c46c693e294784fc1a115"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "c31c809ade349cc2f6d0b7cc05558908"
server_key_iv: "0e11a92e77cb0694f08233997d94ab83"
server_key_version: "encryptionkey@0"
I20260619 20:53:27.185094 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:27.194131 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:27.196350 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:27.196517 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:27.200523 16834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34465
I20260619 20:53:27.200562 16901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34465 every 8 connection(s)
I20260619 20:53:27.200946 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:27.202531 16902 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:53:27.203300 16902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap starting.
I20260619 20:53:27.204182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16834
I20260619 20:53:27.204519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39281
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34465
--webserver_port=35207
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:53:27.212821 16902 log.cc:824] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:27.215454 16902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:53:27.215781 16902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Bootstrap complete.
I20260619 20:53:27.218313 16902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:27.218567 16902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f4db8e2e4c46c693e294784fc1a115, State: Initialized, Role: FOLLOWER
I20260619 20:53:27.218786 16902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:27.218889 16902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:27.218947 16902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:27.219024 16902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:53:27.220064 16902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:27.220191 16902 leader_election.cc:302] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f4db8e2e4c46c693e294784fc1a115; no voters:
I20260619 20:53:27.220419 16902 leader_election.cc:288] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 20:53:27.220458 16906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:53:27.220600 16906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [term 3 LEADER]: Becoming Leader. State: Replica: 48f4db8e2e4c46c693e294784fc1a115, State: Running, Role: LEADER
I20260619 20:53:27.220715 16902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:27.220696 16906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } }
I20260619 20:53:27.221035 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "48f4db8e2e4c46c693e294784fc1a115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } } }
I20260619 20:53:27.221139 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:27.221032 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f4db8e2e4c46c693e294784fc1a115. Latest consensus state: current_term: 3 leader_uuid: "48f4db8e2e4c46c693e294784fc1a115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f4db8e2e4c46c693e294784fc1a115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34465 } } }
I20260619 20:53:27.221359 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:27.221800 16911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:27.222585 16911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:27.222692 16911 catalog_manager.cc:1259] Loaded cluster ID: 5a2389d00e9b448ebf411e3f01aac3c5
I20260619 20:53:27.222723 16911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:27.223877 16911 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:27.224052 16911 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Loaded TSK: 0
I20260619 20:53:27.224098 16911 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48f4db8e2e4c46c693e294784fc1a115: Loaded TSK: 1
F20260619 20:53:27.224349 16911 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1c4b7a19ad google::LogMessage::Fail() at ??:0
@ 0x7f1c4b7a488a google::LogMessage::SendToLog() at ??:0
@ 0x7f1c4b7a14de google::LogMessage::Flush() at ??:0
@ 0x7f1c4b7a2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1c4df0e104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1c4df0bd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1c536fb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1c536fb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1c536fb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1c536c1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1c536cbad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1c4d66f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1c4d664954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1c4e4c86db start_thread at ??:0
@ 0x7f1c4a29f61f clone at ??:0
W20260619 20:53:27.318740 16904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:27.319022 16904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:27.319093 16904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:27.320814 16904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:27.320883 16904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:27.320926 16904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:27.323067 16904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35207
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:27.323320 16904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:27.323606 16904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:27.329052 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:27.328963 16946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:27.328963 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.329700 16904 server_base.cc:1034] running on GCE node
I20260619 20:53:27.329932 16904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:27.330241 16904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:27.331436 16904 hybrid_clock.cc:648] HybridClock initialized: now 1781902407331417 us; error 35 us; skew 500 ppm
Jun 19 20:53:27 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:27.334471 16904 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:53:27.334909 16904 webserver.cc:460] Webserver started at http://127.0.98.1:35207/ using document root <none> and password file <none>
I20260619 20:53:27.335135 16904 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:27.335203 16904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:27.336616 16904 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:27.337409 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.337584 16904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:27.337704 16904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "559c5f912fc6477b985ea848568fb7cc"
format_stamp: "Formatted at 2026-06-19 20:52:45 on dist-test-slave-0smx"
server_key: "e8c3de60401653cd786aada9e0473712"
server_key_iv: "f2fe58eb80f444292d2f78543d8eddc3"
server_key_version: "encryptionkey@0"
I20260619 20:53:27.338045 16904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:27.347797 16904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:27.350139 16904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:27.350301 16904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:27.350523 16904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:27.350986 16960 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:53:27.353035 16904 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:53:27.353096 16904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:53:27.353142 16904 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:53:27.353819 16904 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:53:27.353873 16904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:53:27.353950 16960 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Bootstrap starting.
I20260619 20:53:27.361014 16904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39281
I20260619 20:53:27.361433 16904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:27.365410 16960 log.cc:824] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:27.365466 17067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39281 every 8 connection(s)
I20260619 20:53:27.368147 16960 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:53:27.368482 16960 tablet_bootstrap.cc:492] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Bootstrap complete.
I20260619 20:53:27.369721 16960 ts_tablet_manager.cc:1397] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260619 20:53:27.370056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 16904
I20260619 20:53:27.370313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36109
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34465
--webserver_port=46411
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:53:27.371357 16960 raft_consensus.cc:348] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:27.371690 16960 raft_consensus.cc:729] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559c5f912fc6477b985ea848568fb7cc, State: Initialized, Role: FOLLOWER
I20260619 20:53:27.371863 16960 consensus_queue.cc:260] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:27.372155 16960 ts_tablet_manager.cc:1428] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260619 20:53:27.443703 17070 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:53:27.362113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:54035)
0619 20:53:27.362294 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.362337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:27.362769 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:27.443489 (+ 80720us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":116,"threads_started":1}
W20260619 20:53:27.443967 17068 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34465 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34465: Client connection negotiation failed: client connection to 127.0.98.62:34465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:53:27.444378 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.444249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:45461)
0619 20:53:27.444310 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.444356 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:53:27.444819 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.444716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:33305)
0619 20:53:27.444769 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.444799 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 20:53:27.444960 17068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:53:27.453601 17072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:27.453814 17072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:27.453862 17072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:27.454998 17072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:27.455063 17072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:27.455108 17072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:27.456610 17072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46411
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:27.456899 17072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:27.457130 17072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:27.462164 17078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:27.462128 17081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.462343 17072 server_base.cc:1034] running on GCE node
W20260619 20:53:27.462399 17079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.462574 17072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:27.462780 17072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:27.463992 17072 hybrid_clock.cc:648] HybridClock initialized: now 1781902407463977 us; error 25 us; skew 500 ppm
Jun 19 20:53:27 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:27.466492 17072 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:53:27.466955 17072 webserver.cc:460] Webserver started at http://127.0.98.2:46411/ using document root <none> and password file <none>
I20260619 20:53:27.467250 17072 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:27.467324 17072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:27.468595 17072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:27.469153 17088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.469372 17072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:27.469468 17072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "445ff1991f1c4d5891c8ac77c544ca61"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "77f291f52b2bf1ee55c8673aa19575b5"
server_key_iv: "8547f27e137d42150e927346f0933ccb"
server_key_version: "encryptionkey@0"
I20260619 20:53:27.469825 17072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:27.496459 17072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:27.498898 17072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:27.499066 17072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:27.499317 17072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:27.499622 17072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:27.499675 17072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.499720 17072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:27.499752 17072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.505836 17072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36109
I20260619 20:53:27.505894 17201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36109 every 8 connection(s)
I20260619 20:53:27.506266 17072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:27.507320 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.506932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:44653)
0619 20:53:27.507078 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.507174 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
W20260619 20:53:27.507714 17202 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34465 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34465: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
I20260619 20:53:27.507970 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.507908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:56239)
0619 20:53:27.507933 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.507956 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260619 20:53:27.508219 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.508170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:50955)
0619 20:53:27.508188 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.508207 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260619 20:53:27.508347 17202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:53:27.515973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17072
I20260619 20:53:27.516204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34695
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34465
--webserver_port=40197
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:27.597564 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:27.597831 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:27.597873 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:27.599048 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:27.599128 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:27.599174 17205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:27.600622 17205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40197
--tserver_master_addrs=127.0.98.62:34465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:27.600898 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:27.601150 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:27.606556 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.606593 17205 server_base.cc:1034] running on GCE node
W20260619 20:53:27.606575 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:27.606575 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:27.606967 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:27.607218 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:27.608367 17205 hybrid_clock.cc:648] HybridClock initialized: now 1781902407608344 us; error 39 us; skew 500 ppm
Jun 19 20:53:27 dist-test-slave-0smx krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:27.610998 17205 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:53:27.611495 17205 webserver.cc:460] Webserver started at http://127.0.98.3:40197/ using document root <none> and password file <none>
I20260619 20:53:27.611717 17205 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:27.611783 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:27.613219 17205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:27.613828 17220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.614001 17205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.614099 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "89f5de920cdb4872b35dad447402a3dc"
format_stamp: "Formatted at 2026-06-19 20:52:46 on dist-test-slave-0smx"
server_key: "f02c19d73151fee0b2e3a1aa7cdd8389"
server_key_iv: "dc996e7a70751b27fc5010cfd2b0761d"
server_key_version: "encryptionkey@0"
I20260619 20:53:27.614467 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:27.626412 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:27.628749 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:27.628913 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:27.629163 17205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:27.629516 17205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:27.629570 17205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.629612 17205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:27.629637 17205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:27.635573 17205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34695
I20260619 20:53:27.635620 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34695 every 8 connection(s)
I20260619 20:53:27.635970 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:53:27.636936 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.636574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:53243)
0619 20:53:27.636728 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.636796 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
W20260619 20:53:27.637401 17334 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34465 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34465: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
I20260619 20:53:27.637686 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.637602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:60839)
0619 20:53:27.637645 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.637669 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 20:53:27.637900 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:27.637854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:59723)
0619 20:53:27.637872 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:27.637889 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260619 20:53:27.638029 17334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:53:27.641037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17205
I20260619 20:53:28.445679 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.445451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:33797)
0619 20:53:28.445590 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:28.445628 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260619 20:53:28.508993 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.508727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:48381)
0619 20:53:28.508913 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:28.508949 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":53,"threads_started":1}
I20260619 20:53:28.638634 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.638393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:60773)
0619 20:53:28.638543 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:28.638584 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260619 20:53:28.876302 17341 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:28.876415 17341 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:28.876811 17341 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
Jun 19 20:53:28 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 20:53:28 dist-test-slave-0smx krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 20:53:28.887130 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.876968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.98.1:58993)
0619 20:53:28.877024 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:28.877036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:28.877093 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:28.878033 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:28.878036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:28.878412 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:28.878598 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:28.878604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.879711 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.879713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:28.880498 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:28.880501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.881249 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.881250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:28.881339 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:28.881889 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:28.881897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:28.883293 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:28.885443 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:28.885449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:28.885451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:28.885713 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:28.885966 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:28.885967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:28.885967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:28.886058 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:28.886532 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:28.886537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:28.886695 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 20:53:28.886851 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 20:53:28.887027 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":80}
I20260619 20:53:28.887154 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.877065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58993 (local address 127.0.98.2:36109)
0619 20:53:28.877283 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:28.877332 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:28.877373 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:53:28.877804 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:28.877807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:28.877846 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:28.877998 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:28.878728 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.879584 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.880594 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.881151 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.881278 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:28.883419 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:28.883425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:28.883425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:28.883437 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:28.885364 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:28.885798 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:28.885800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:28.885800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:28.885879 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:28.886124 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:28.886125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:28.886126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:28.886462 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:28.886548 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:28.886907 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 20:53:28.887041 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":53,"threads_started":1}
I20260619 20:53:28.887154 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.877166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52601 (local address 127.0.98.3:34695)
0619 20:53:28.877324 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:28.877335 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:28.877365 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:53:28.877804 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:28.877806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:28.877847 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:28.877997 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:28.878728 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.879591 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.880590 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.881125 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.881222 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:28.883706 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:28.883710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:28.883710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:28.883722 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:28.885603 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:28.885978 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:28.885981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:28.885981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:28.886066 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:28.886265 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:28.886266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:28.886266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:28.886530 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:28.886600 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:28.886907 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 20:53:28.887041 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":85,"threads_started":1}
I20260619 20:53:28.887267 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:28.876979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34695 (local address 127.0.98.1:52601)
0619 20:53:28.877109 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:28.877120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:28.877172 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:28.878033 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:28.878035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:28.878412 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:28.878598 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:28.878604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.879746 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.879747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:28.880498 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:28.880500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:28.881207 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:28.881208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:28.881335 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:28.882058 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:28.882065 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:28.883605 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:28.885677 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:28.885678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:28.885679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:28.885872 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:28.886157 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:28.886158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:28.886158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:28.886204 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:28.886595 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:28.886596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:28.886695 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 20:53:28.886830 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 20:53:28.887168 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":195,"thread_start_us":59,"threads_started":1}
W20260619 20:53:28.887957 16957 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
W20260619 20:53:28.888026 16955 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
I20260619 20:53:28.888053 16957 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 559c5f912fc6477b985ea848568fb7cc; no voters: 445ff1991f1c4d5891c8ac77c544ca61, 89f5de920cdb4872b35dad447402a3dc
I20260619 20:53:28.888163 17341 raft_consensus.cc:2726] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:53:29.446394 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:29.446185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:40031)
0619 20:53:29.446312 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:29.446349 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260619 20:53:29.509680 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:29.509504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:32917)
0619 20:53:29.509617 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:29.509645 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260619 20:53:29.639303 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:29.639114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:35171)
0619 20:53:29.639229 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:29.639266 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260619 20:53:30.447175 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:30.446941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:57553)
0619 20:53:30.447070 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:30.447112 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260619 20:53:30.510319 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:30.510126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:59507)
0619 20:53:30.510253 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:30.510284 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260619 20:53:30.639905 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:30.639730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:49475)
0619 20:53:30.639837 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:30.639869 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260619 20:53:30.695348 17351 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:30.695468 17351 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:30.695683 17351 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
W20260619 20:53:30.696084 16957 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
W20260619 20:53:30.696185 16955 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
I20260619 20:53:30.696286 16955 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 559c5f912fc6477b985ea848568fb7cc; no voters: 445ff1991f1c4d5891c8ac77c544ca61, 89f5de920cdb4872b35dad447402a3dc
I20260619 20:53:30.696386 17351 raft_consensus.cc:2726] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:53:31.447860 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:31.447650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:54197)
0619 20:53:31.447773 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:31.447814 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260619 20:53:31.510993 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:31.510799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:59405)
0619 20:53:31.510916 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:31.510953 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260619 20:53:31.640535 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:31.640287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:51403)
0619 20:53:31.640451 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:31.640495 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
I20260619 20:53:32.448578 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:32.448368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:42991)
0619 20:53:32.448505 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:32.448533 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260619 20:53:32.511709 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:32.511515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:45007)
0619 20:53:32.511642 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:32.511675 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260619 20:53:32.641460 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:32.641115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:60579)
0619 20:53:32.641390 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:32.641426 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":212,"threads_started":1}
I20260619 20:53:32.996512 17358 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:32.996614 17358 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:32.996774 17358 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
W20260619 20:53:32.997234 16955 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
W20260619 20:53:32.997334 16957 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
I20260619 20:53:32.997380 16957 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 559c5f912fc6477b985ea848568fb7cc; no voters: 445ff1991f1c4d5891c8ac77c544ca61, 89f5de920cdb4872b35dad447402a3dc
I20260619 20:53:32.997490 17358 raft_consensus.cc:2726] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:53:33.449533 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:33.449252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:45645)
0619 20:53:33.449440 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:33.449482 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":113,"threads_started":1}
I20260619 20:53:33.512415 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:33.512191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:37741)
0619 20:53:33.512321 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:33.512363 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260619 20:53:33.642266 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:33.642050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:55151)
0619 20:53:33.642187 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:33.642222 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
I20260619 20:53:34.450163 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:34.449958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:49057)
0619 20:53:34.450085 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:34.450118 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260619 20:53:34.513123 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:34.512914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:34515)
0619 20:53:34.513046 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:34.513081 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260619 20:53:34.642939 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:34.642684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:58259)
0619 20:53:34.642856 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:34.642896 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260619 20:53:35.450809 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:35.450572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:59547)
0619 20:53:35.450725 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:35.450764 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260619 20:53:35.513793 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:35.513563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:48447)
0619 20:53:35.513683 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:35.513734 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260619 20:53:35.643568 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:35.643364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:40321)
0619 20:53:35.643500 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:35.643529 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260619 20:53:36.451471 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:36.451234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:33953)
0619 20:53:36.451372 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:36.451408 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260619 20:53:36.514526 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:36.514302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:43169)
0619 20:53:36.514449 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:36.514483 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260619 20:53:36.644397 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:36.644197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:59487)
0619 20:53:36.644305 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:36.644355 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260619 20:53:36.891996 17371 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:36.892086 17371 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:36.892290 17371 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
W20260619 20:53:36.892668 16955 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
W20260619 20:53:36.892724 16957 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
I20260619 20:53:36.892766 16957 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 559c5f912fc6477b985ea848568fb7cc; no voters: 445ff1991f1c4d5891c8ac77c544ca61, 89f5de920cdb4872b35dad447402a3dc
I20260619 20:53:36.892843 17371 raft_consensus.cc:2726] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:53:37.452169 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:37.451966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:57619)
0619 20:53:37.452091 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:37.452124 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260619 20:53:37.515137 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:37.514947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:45053)
0619 20:53:37.515060 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:37.515096 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260619 20:53:37.645246 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:37.644978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:33083)
0619 20:53:37.645157 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:37.645189 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":124,"threads_started":1}
I20260619 20:53:38.452829 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:38.452634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:45133)
0619 20:53:38.452753 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:38.452784 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260619 20:53:38.515753 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:38.515541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:54507)
0619 20:53:38.515663 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:38.515714 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260619 20:53:38.645867 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:38.645678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:38699)
0619 20:53:38.645793 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:38.645825 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260619 20:53:38.857098 17378 raft_consensus.cc:482] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:38.857195 17378 raft_consensus.cc:504] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f5de920cdb4872b35dad447402a3dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34695 } } peers { permanent_uuid: "445ff1991f1c4d5891c8ac77c544ca61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "559c5f912fc6477b985ea848568fb7cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39281 } }
I20260619 20:53:38.857477 17378 leader_election.cc:288] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695), 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109)
W20260619 20:53:38.857867 16955 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 445ff1991f1c4d5891c8ac77c544ca61 (127.0.98.2:36109): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
W20260619 20:53:38.857980 16957 leader_election.cc:341] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89f5de920cdb4872b35dad447402a3dc (127.0.98.3:34695): Not found: Tablet not found: 1ffa3da600834c3383b68a264c15e030
I20260619 20:53:38.858048 16957 leader_election.cc:302] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 559c5f912fc6477b985ea848568fb7cc; no voters: 445ff1991f1c4d5891c8ac77c544ca61, 89f5de920cdb4872b35dad447402a3dc
I20260619 20:53:38.858147 17378 raft_consensus.cc:2726] T 1ffa3da600834c3383b68a264c15e030 P 559c5f912fc6477b985ea848568fb7cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:53:39.453605 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:39.453399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:55971)
0619 20:53:39.453521 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:39.453557 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260619 20:53:39.516431 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:39.516219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:53619)
0619 20:53:39.516354 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:39.516391 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260619 20:53:39.646433 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:39.646248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:38009)
0619 20:53:39.646360 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:39.646393 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260619 20:53:40.454227 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:40.453997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:43215)
0619 20:53:40.454147 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:40.454182 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260619 20:53:40.517094 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:40.516918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:44461)
0619 20:53:40.517016 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:40.517050 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
I20260619 20:53:40.647045 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:40.646837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:34449)
0619 20:53:40.646963 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:40.646998 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260619 20:53:41.455008 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:41.454714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:36737)
0619 20:53:41.454928 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:41.454964 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":127,"threads_started":1}
I20260619 20:53:41.517788 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:41.517568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:40799)
0619 20:53:41.517709 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:41.517746 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260619 20:53:41.647794 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:41.647583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:47445)
0619 20:53:41.647717 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:41.647751 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260619 20:53:42.455705 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:42.455505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.1:52095)
0619 20:53:42.455631 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:42.455662 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260619 20:53:42.518481 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:42.518273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.2:43063)
0619 20:53:42.518405 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:42.518439 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260619 20:53:42.648600 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:42.648399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34465 (local address 127.0.98.3:60871)
0619 20:53:42.648515 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:42.648556 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260619 20:53:42.649788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 16904
I20260619 20:53:42.655383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17072
I20260619 20:53:42.659650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17205
2026-06-19T20:53:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58147 ms)
[----------] 30 tests from SecurityITest (256300 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:42 dist-test-slave-0smx krb5kdc[17393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:42 dist-test-slave-0smx krb5kdc[17393](info): set up 2 sockets
Jun 19 20:53:42 dist-test-slave-0smx krb5kdc[17393](info): commencing operation
krb5kdc: starting...
W20260619 20:53:43.692945 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:43 dist-test-slave-0smx krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:53:43.804082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:43.882505 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:43.882726 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:43.882761 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:43.883886 17406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:43.883949 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:43.883993 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:43.884027 17406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:43.884049 17406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:43.885512 17406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:43.885759 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:43.885982 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:53:43.890791 17406 server_base.cc:1034] running on GCE node
W20260619 20:53:43.890851 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:43.890796 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:43.891006 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:43.891307 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:43.891628 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:43.892800 17406 hybrid_clock.cc:648] HybridClock initialized: now 1781902423892795 us; error 33 us; skew 500 ppm
Jun 19 20:53:43 dist-test-slave-0smx krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:43.898452 17406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:53:43.898885 17406 webserver.cc:460] Webserver started at http://127.0.98.62:40633/ using document root <none> and password file <none>
I20260619 20:53:43.899128 17406 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:43.899173 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:43.899358 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:43.900409 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "e133b7a35d044fdc8efb77a15054deff"
format_stamp: "Formatted at 2026-06-19 20:53:43 on dist-test-slave-0smx"
server_key: "a63277143a628bdd0129eb561ca15dbe"
server_key_iv: "8d0f7fd04b12a5f720f93cb52dd533ca"
server_key_version: "encryptionkey@0"
I20260619 20:53:43.900764 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "e133b7a35d044fdc8efb77a15054deff"
format_stamp: "Formatted at 2026-06-19 20:53:43 on dist-test-slave-0smx"
server_key: "a63277143a628bdd0129eb561ca15dbe"
server_key_iv: "8d0f7fd04b12a5f720f93cb52dd533ca"
server_key_version: "encryptionkey@0"
I20260619 20:53:43.902122 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:43.902822 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:43.902997 17406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:43.903086 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "e133b7a35d044fdc8efb77a15054deff"
format_stamp: "Formatted at 2026-06-19 20:53:43 on dist-test-slave-0smx"
server_key: "a63277143a628bdd0129eb561ca15dbe"
server_key_iv: "8d0f7fd04b12a5f720f93cb52dd533ca"
server_key_version: "encryptionkey@0"
I20260619 20:53:43.903167 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:43.919787 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:43.921556 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:43.921720 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:43.925519 17406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45019
I20260619 20:53:43.925549 17473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45019 every 8 connection(s)
I20260619 20:53:43.925887 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:43.926746 17474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:43.928434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17406
I20260619 20:53:43.928543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:43.928726 392 external_mini_cluster.cc:1428] Setting key 8c185d3e1048a1f72b03c17c368b7794
I20260619 20:53:43.929292 17474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff: Bootstrap starting.
I20260619 20:53:43.929908 17474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:43.930238 17474 log.cc:824] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:43.930970 17474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff: No bootstrap required, opened a new log
I20260619 20:53:43.933107 17474 raft_consensus.cc:348] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133b7a35d044fdc8efb77a15054deff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260619 20:53:43.933235 17474 raft_consensus.cc:374] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:43.933257 17474 raft_consensus.cc:729] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e133b7a35d044fdc8efb77a15054deff, State: Initialized, Role: FOLLOWER
I20260619 20:53:43.933413 17474 consensus_queue.cc:260] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133b7a35d044fdc8efb77a15054deff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260619 20:53:43.933490 17474 raft_consensus.cc:388] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:43.933512 17474 raft_consensus.cc:482] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:43.933578 17474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:43.934170 17474 raft_consensus.cc:504] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133b7a35d044fdc8efb77a15054deff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260619 20:53:43.934306 17474 leader_election.cc:302] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e133b7a35d044fdc8efb77a15054deff; no voters:
I20260619 20:53:43.934484 17474 leader_election.cc:288] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:43.934509 17479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:43.934630 17479 raft_consensus.cc:686] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [term 1 LEADER]: Becoming Leader. State: Replica: e133b7a35d044fdc8efb77a15054deff, State: Running, Role: LEADER
I20260619 20:53:43.934733 17474 sys_catalog.cc:564] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:43.934764 17479 consensus_queue.cc:237] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133b7a35d044fdc8efb77a15054deff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260619 20:53:43.935207 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e133b7a35d044fdc8efb77a15054deff. Latest consensus state: current_term: 1 leader_uuid: "e133b7a35d044fdc8efb77a15054deff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133b7a35d044fdc8efb77a15054deff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } }
I20260619 20:53:43.935220 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e133b7a35d044fdc8efb77a15054deff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e133b7a35d044fdc8efb77a15054deff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } }
I20260619 20:53:43.935401 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:43.935413 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:43.935638 17488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 20:53:43 dist-test-slave-0smx krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:43.936182 17488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:43.939649 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:43.929277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.0.98.62:45019)
0619 20:53:43.929521 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:43.929530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:43.929562 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:53:43.930040 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:43.930047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:43.930093 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:43.930200 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:43.930956 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:43.931889 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:43.933032 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:43.933608 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:43.933725 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:43.936657 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:43.936661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:43.936661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:43.936673 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:43.938217 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:43.938578 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:43.938581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:43.938582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:43.938668 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:43.938883 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:43.938884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:43.938884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:43.939103 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:43.939276 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:43.939365 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:53:43.939493 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260619 20:53:43.941684 17488 catalog_manager.cc:1347] Generated new cluster ID: 79ad0972ff524fad9ffb6a05258a55b7
I20260619 20:53:43.941740 17488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:43.954906 17488 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:43.955596 17488 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:43.964910 17488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff: Generated new TSK 0
I20260619 20:53:43.965086 17488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:53:43.995816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:44.082386 17500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:44.082609 17500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:44.082649 17500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:44.083847 17500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:44.083909 17500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:44.083974 17500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:44.085513 17500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:44.085762 17500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:44.085979 17500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:44.090811 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.090811 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.090808 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.090937 17500 server_base.cc:1034] running on GCE node
I20260619 20:53:44.091244 17500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:44.091478 17500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:44.092615 17500 hybrid_clock.cc:648] HybridClock initialized: now 1781902424092595 us; error 36 us; skew 500 ppm
Jun 19 20:53:44 dist-test-slave-0smx krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:44.095139 17500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:53:44.095549 17500 webserver.cc:460] Webserver started at http://127.0.98.1:46057/ using document root <none> and password file <none>
I20260619 20:53:44.095769 17500 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:44.095811 17500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:44.095961 17500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:44.096966 17500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "b05144c5f4cb4e2bae2b21505763849a"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "ee059be5662666b145782dd154bcac8e"
server_key_iv: "67d1aa9d007507f1fcfd273bc4baa568"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.097364 17500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b05144c5f4cb4e2bae2b21505763849a"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "ee059be5662666b145782dd154bcac8e"
server_key_iv: "67d1aa9d007507f1fcfd273bc4baa568"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.098635 17500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:44.099316 17515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.099493 17500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:53:44.099555 17500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "b05144c5f4cb4e2bae2b21505763849a"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "ee059be5662666b145782dd154bcac8e"
server_key_iv: "67d1aa9d007507f1fcfd273bc4baa568"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.099637 17500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:44.108659 17500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:44.110891 17500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:44.111047 17500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:44.111261 17500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:44.111541 17500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:44.111574 17500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.111593 17500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:44.111605 17500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.117364 17500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37547
I20260619 20:53:44.117411 17628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37547 every 8 connection(s)
I20260619 20:53:44.117717 17500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:44.120749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17500
I20260619 20:53:44.120877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:53:44.121078 392 external_mini_cluster.cc:1428] Setting key c42fb1cf4c0c4c9b6f5207fb7e9686a4
Jun 19 20:53:44 dist-test-slave-0smx krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:44.127645 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.118393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35093 (local address 127.0.98.62:45019)
0619 20:53:44.118471 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.118473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.118675 (+ 202us) server_negotiation.cc:407] Connection header received
0619 20:53:44.119594 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.119595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.119607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.119672 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.120269 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.120781 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.122076 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.122295 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.122377 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.124469 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.124470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.124471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:44.124479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.126281 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.126681 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.126682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.126683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.126747 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.126981 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.126982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.126982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.127129 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.127204 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.127444 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.127540 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 20:53:44.127661 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.118384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.1:35093)
0619 20:53:44.118557 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.118591 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.119485 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.119743 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.119747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.119992 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:44.120174 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.120177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.120875 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.120876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.121978 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.121981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.122386 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.122387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.122515 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.123037 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:44.123046 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:44.124376 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:44.126364 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:44.126369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:44.126371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:44.126585 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:44.126822 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:44.126823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:44.126823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:44.126915 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:44.127197 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:44.127201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:44.127312 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.127393 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.127510 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260619 20:53:44.128194 17629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260619 20:53:44.128291 17629 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:44.128480 17629 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260619 20:53:44.129024 17438 ts_manager.cc:194] Registered new tserver with Master: b05144c5f4cb4e2bae2b21505763849a (127.0.98.1:37547)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:53:44.129742 17438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:53:44.146162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:44.226136 17634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:44.226331 17634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:44.226353 17634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:44.227427 17634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:44.227456 17634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:44.227480 17634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:44.228895 17634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:44.229077 17634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:44.229245 17634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:44.234212 17642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.234220 17640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.234226 17639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.234449 17634 server_base.cc:1034] running on GCE node
I20260619 20:53:44.234731 17634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:44.235075 17634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:44.236258 17634 hybrid_clock.cc:648] HybridClock initialized: now 1781902424236246 us; error 40 us; skew 500 ppm
Jun 19 20:53:44 dist-test-slave-0smx krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:44.238881 17634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:53:44.239300 17634 webserver.cc:460] Webserver started at http://127.0.98.2:35899/ using document root <none> and password file <none>
I20260619 20:53:44.239535 17634 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:44.239605 17634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:44.239754 17634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:44.240841 17634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c00518a0a884679a6116680ff6de161"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "35ef35347e1ea829852a8586d5d02cd1"
server_key_iv: "a19c0500a1e0e233cbe0ee404ab8682b"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.241204 17634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c00518a0a884679a6116680ff6de161"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "35ef35347e1ea829852a8586d5d02cd1"
server_key_iv: "a19c0500a1e0e233cbe0ee404ab8682b"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.242539 17634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:44.243252 17649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.243427 17634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.243523 17634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "1c00518a0a884679a6116680ff6de161"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "35ef35347e1ea829852a8586d5d02cd1"
server_key_iv: "a19c0500a1e0e233cbe0ee404ab8682b"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.243605 17634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:44.260758 17634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:44.263224 17634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:44.263394 17634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:44.263641 17634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:44.263964 17634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:44.264024 17634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.264065 17634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:44.264093 17634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.269518 17634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43845
I20260619 20:53:44.269606 17762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43845 every 8 connection(s)
I20260619 20:53:44.269923 17634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:44.270689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17634
I20260619 20:53:44.270802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:53:44.271016 392 external_mini_cluster.cc:1428] Setting key 1fc51f1e54348203af00afacfffa06fb
Jun 19 20:53:44 dist-test-slave-0smx krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 20:53:44.280692 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.270690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49087 (local address 127.0.98.62:45019)
0619 20:53:44.270760 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.270762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.270928 (+ 166us) server_negotiation.cc:407] Connection header received
0619 20:53:44.272048 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.272049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.272061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.272141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.272725 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.273484 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.275051 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.275239 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.275322 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.277855 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.277856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.277856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:44.277863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.279233 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.279721 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.279722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.279722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.279771 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.280062 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.280062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.280063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.280201 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.280256 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.280562 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.280630 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:53:44.280833 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.270643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.2:49087)
0619 20:53:44.270793 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.270834 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.271932 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.272196 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.272199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.272449 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:44.272636 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.272640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.273589 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.273590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.274968 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.274971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.275347 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.275348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.275522 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.276028 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:44.276038 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:44.277757 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:44.279312 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:44.279319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:44.279321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:44.279620 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:44.279843 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:44.279844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:44.279844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:44.279999 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:44.280273 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:44.280279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:44.280428 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.280535 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.280645 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260619 20:53:44.281375 17763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260619 20:53:44.281488 17763 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:44.281664 17763 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260619 20:53:44.282075 17438 ts_manager.cc:194] Registered new tserver with Master: 1c00518a0a884679a6116680ff6de161 (127.0.98.2:43845)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 20:53:44.282500 17438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:53:44.296211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:44.379068 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:44.379263 17768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:44.379288 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:44.380462 17768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:44.380527 17768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:44.380555 17768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:44.382097 17768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:44.382352 17768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:44.382586 17768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:44.387570 17774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.387575 17773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.387599 17768 server_base.cc:1034] running on GCE node
W20260619 20:53:44.387593 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.388089 17768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:44.388331 17768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:44.389464 17768 hybrid_clock.cc:648] HybridClock initialized: now 1781902424389457 us; error 29 us; skew 500 ppm
Jun 19 20:53:44 dist-test-slave-0smx krb5kdc[17393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:44.391891 17768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:53:44.392259 17768 webserver.cc:460] Webserver started at http://127.0.98.3:35839/ using document root <none> and password file <none>
I20260619 20:53:44.392437 17768 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:44.392477 17768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:44.392565 17768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:44.393566 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "55a7dc6d6cae45aaa288589d1ac09042"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "70721600a8d7297fe0343dfde3f8b9da"
server_key_iv: "8d95de5a5bc65d50e50a38bd58e7ccc5"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.393855 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55a7dc6d6cae45aaa288589d1ac09042"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "70721600a8d7297fe0343dfde3f8b9da"
server_key_iv: "8d95de5a5bc65d50e50a38bd58e7ccc5"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.395083 17768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:44.395728 17783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.395884 17768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:44.395977 17768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "55a7dc6d6cae45aaa288589d1ac09042"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "70721600a8d7297fe0343dfde3f8b9da"
server_key_iv: "8d95de5a5bc65d50e50a38bd58e7ccc5"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.396059 17768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:44.403812 17768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:44.406008 17768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:44.406116 17768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:44.406255 17768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:44.406526 17768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:44.406585 17768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.406623 17768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:44.406649 17768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.412352 17768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38949
I20260619 20:53:44.412420 17896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38949 every 8 connection(s)
I20260619 20:53:44.412719 17768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:53:44 dist-test-slave-0smx krb5kdc[17393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:44.420874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17768
I20260619 20:53:44.420981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:53:44.421168 392 external_mini_cluster.cc:1428] Setting key 5a583c2a82fd0355ca1e17d7c9d293f0
I20260619 20:53:44.421864 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.413514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54845 (local address 127.0.98.62:45019)
0619 20:53:44.413576 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.413578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.413783 (+ 205us) server_negotiation.cc:407] Connection header received
0619 20:53:44.414742 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.414743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.414758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.414808 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.415407 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.415919 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.416926 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.417132 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.417216 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.419283 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.419285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.419285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:44.419292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.420419 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.420825 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.420827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.420827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.420883 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.421156 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.421157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.421157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.421302 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.421379 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.421702 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.421778 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:53:44.422049 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.413493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.3:54845)
0619 20:53:44.413659 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.413700 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.414625 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.414875 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.414879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.415132 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:44.415313 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.415317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.416032 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.416034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.416835 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.416837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.417227 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.417228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.417435 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.417890 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:44.417897 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:44.419188 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:44.420523 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:44.420530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:44.420532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:44.420738 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:44.420949 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:44.420950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:44.420950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:44.421095 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:44.421383 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:44.421390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:44.421549 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.421667 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.421814 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260619 20:53:44.422513 17897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260619 20:53:44.422602 17897 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:44.422781 17897 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260619 20:53:44.423141 17438 ts_manager.cc:194] Registered new tserver with Master: 55a7dc6d6cae45aaa288589d1ac09042 (127.0.98.3:38949)
I20260619 20:53:44.423471 17438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54845
I20260619 20:53:44.432854 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:44.440239 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.434200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.0.98.62:45019)
0619 20:53:44.434237 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.434238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.434375 (+ 137us) server_negotiation.cc:407] Connection header received
0619 20:53:44.434448 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.434449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.434458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.434525 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.435115 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.435672 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.436480 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.436670 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.436741 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.437991 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.437992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.437993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:44.437999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.439127 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.439448 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.439449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.439449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.439504 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:44.439715 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:44.439716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:44.439716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:44.439852 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.439959 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.440069 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.440146 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:53:44.441617 17438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:44.443085 17438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:44.449203 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.445066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.0.98.3:38949)
0619 20:53:44.445455 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.445460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.445471 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:53:44.445506 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.445509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.445525 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.445613 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:44.445828 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.446979 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448218 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.448854 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.448992 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.449089 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.449122 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":74,"threads_started":1}
I20260619 20:53:44.449237 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.444969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38949 (local address 127.0.0.1:45338)
0619 20:53:44.445091 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.445134 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.445209 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.445623 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.445625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.445627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:44.445693 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.445695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.447076 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.447077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.448146 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.448148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.448956 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.449018 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.449022 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.449077 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.449120 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":48,"thread_start_us":64,"threads_started":1}
I20260619 20:53:44.449326 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.445366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.98.1:37547)
0619 20:53:44.445535 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.445539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.445554 (+ 15us) server_negotiation.cc:407] Connection header received
0619 20:53:44.445602 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.445604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.445619 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.445695 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:44.446037 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.447106 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448059 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.448741 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448816 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.448874 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.449196 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.449235 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260619 20:53:44.449118 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.445164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37547 (local address 127.0.0.1:36462)
0619 20:53:44.445369 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.445378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.445420 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.445780 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.445781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.445783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:44.445958 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.445960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.447208 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.447209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.447969 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.447971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.448827 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.448900 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.448914 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.448977 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.449025 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":46,"threads_started":1}
I20260619 20:53:44.449368 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.445218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.0.98.2:43845)
0619 20:53:44.445590 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.445594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.445602 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:44.445640 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.445642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.445655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.445726 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:44.445836 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.446745 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448385 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.448899 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.449158 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.449223 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.449244 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.449278 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":110,"threads_started":1}
I20260619 20:53:44.449543 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.444987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43845 (local address 127.0.0.1:40006)
0619 20:53:44.445242 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.445254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.445321 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.445725 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.445726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.445728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:44.445782 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.445784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.446852 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.446853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.448304 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.448305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.448994 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.448994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.449043 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.449046 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.449079 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.449113 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":62,"threads_started":1}
I20260619 20:53:44.450106 17563 tablet_service.cc:1467] Processing CreateTablet for tablet 39e8d0327d0440eab8bd4bf8169bd132 (DEFAULT_TABLE table=test-table [id=0d8c3203e5414189b4ca92ff9641ecf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:44.450212 17697 tablet_service.cc:1467] Processing CreateTablet for tablet 39e8d0327d0440eab8bd4bf8169bd132 (DEFAULT_TABLE table=test-table [id=0d8c3203e5414189b4ca92ff9641ecf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:44.450449 17563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e8d0327d0440eab8bd4bf8169bd132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:44.450546 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e8d0327d0440eab8bd4bf8169bd132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:44.450502 17831 tablet_service.cc:1467] Processing CreateTablet for tablet 39e8d0327d0440eab8bd4bf8169bd132 (DEFAULT_TABLE table=test-table [id=0d8c3203e5414189b4ca92ff9641ecf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:44.450763 17831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e8d0327d0440eab8bd4bf8169bd132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:44.453097 17911 tablet_bootstrap.cc:492] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Bootstrap starting.
I20260619 20:53:44.453617 17912 tablet_bootstrap.cc:492] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: Bootstrap starting.
I20260619 20:53:44.453780 17911 tablet_bootstrap.cc:654] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:44.453816 17913 tablet_bootstrap.cc:492] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Bootstrap starting.
I20260619 20:53:44.454385 17911 log.cc:824] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:44.454434 17913 tablet_bootstrap.cc:654] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:44.454960 17913 log.cc:824] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:44.455132 17911 tablet_bootstrap.cc:492] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: No bootstrap required, opened a new log
I20260619 20:53:44.455206 17911 ts_tablet_manager.cc:1397] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:44.455682 17913 tablet_bootstrap.cc:492] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: No bootstrap required, opened a new log
I20260619 20:53:44.455767 17913 ts_tablet_manager.cc:1397] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:44.455816 17912 tablet_bootstrap.cc:654] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:44.456446 17912 log.cc:824] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:44.457218 17912 tablet_bootstrap.cc:492] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: No bootstrap required, opened a new log
I20260619 20:53:44.457310 17912 ts_tablet_manager.cc:1397] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 20:53:44.457602 17911 raft_consensus.cc:348] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.457789 17911 raft_consensus.cc:374] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:44.457686 17913 raft_consensus.cc:348] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.457829 17911 raft_consensus.cc:729] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c00518a0a884679a6116680ff6de161, State: Initialized, Role: FOLLOWER
I20260619 20:53:44.457835 17913 raft_consensus.cc:374] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:44.457870 17913 raft_consensus.cc:729] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a7dc6d6cae45aaa288589d1ac09042, State: Initialized, Role: FOLLOWER
I20260619 20:53:44.457986 17911 consensus_queue.cc:260] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.457996 17913 consensus_queue.cc:260] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.458261 17913 ts_tablet_manager.cc:1428] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:44.458266 17911 ts_tablet_manager.cc:1428] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:53:44.458401 17897 heartbeater.cc:503] Master 127.0.98.62:45019 was elected leader, sending a full tablet report...
I20260619 20:53:44.458648 17763 heartbeater.cc:503] Master 127.0.98.62:45019 was elected leader, sending a full tablet report...
I20260619 20:53:44.459407 17912 raft_consensus.cc:348] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.459542 17912 raft_consensus.cc:374] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:44.459576 17912 raft_consensus.cc:729] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05144c5f4cb4e2bae2b21505763849a, State: Initialized, Role: FOLLOWER
I20260619 20:53:44.459676 17912 consensus_queue.cc:260] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.459872 17912 ts_tablet_manager.cc:1428] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:53:44.459926 17629 heartbeater.cc:503] Master 127.0.98.62:45019 was elected leader, sending a full tablet report...
I20260619 20:53:44.493389 17918 raft_consensus.cc:482] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:44.493516 17918 raft_consensus.cc:504] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.493925 17918 leader_election.cc:288] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c00518a0a884679a6116680ff6de161 (127.0.98.2:43845), b05144c5f4cb4e2bae2b21505763849a (127.0.98.1:37547)
I20260619 20:53:44.497692 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.494158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55235 (local address 127.0.98.1:37547)
0619 20:53:44.494199 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.494201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.494379 (+ 178us) server_negotiation.cc:407] Connection header received
0619 20:53:44.494401 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.494402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.494413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.494469 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:44.494614 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.495261 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.496753 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.497339 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.497409 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.497419 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.497575 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.497610 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:53:44.497692 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.494135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40203 (local address 127.0.98.2:43845)
0619 20:53:44.494181 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.494182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.494193 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:53:44.494238 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.494239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.494257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.494298 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:44.494498 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.495227 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.496753 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.497339 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.497409 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.497419 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.497585 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.497620 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:53:44.497746 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.494066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43845 (local address 127.0.98.3:40203)
0619 20:53:44.494119 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.494130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.494171 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.494322 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.494323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.494325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:44.494401 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.494403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.495333 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.495333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.496667 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.496670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.497416 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.497417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.497497 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.497502 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.497559 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.497611 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":81}
I20260619 20:53:44.497756 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.494111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37547 (local address 127.0.98.3:55235)
0619 20:53:44.494272 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.494294 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.494345 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.494486 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.494487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.494489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:44.494549 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.494550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.495330 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.495331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.496667 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.496670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.497416 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.497417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.497497 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.497502 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.497559 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.497611 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":57,"thread_start_us":94,"threads_started":1}
I20260619 20:53:44.498122 17717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8d0327d0440eab8bd4bf8169bd132" candidate_uuid: "55a7dc6d6cae45aaa288589d1ac09042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c00518a0a884679a6116680ff6de161" is_pre_election: true
I20260619 20:53:44.498221 17583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8d0327d0440eab8bd4bf8169bd132" candidate_uuid: "55a7dc6d6cae45aaa288589d1ac09042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05144c5f4cb4e2bae2b21505763849a" is_pre_election: true
I20260619 20:53:44.498282 17717 raft_consensus.cc:2454] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a7dc6d6cae45aaa288589d1ac09042 in term 0.
I20260619 20:53:44.498339 17583 raft_consensus.cc:2454] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a7dc6d6cae45aaa288589d1ac09042 in term 0.
I20260619 20:53:44.498528 17787 leader_election.cc:302] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c00518a0a884679a6116680ff6de161, 55a7dc6d6cae45aaa288589d1ac09042; no voters:
I20260619 20:53:44.498648 17918 raft_consensus.cc:2781] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:44.498721 17918 raft_consensus.cc:482] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:44.498739 17918 raft_consensus.cc:3037] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:44.499439 17918 raft_consensus.cc:504] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.499599 17918 leader_election.cc:288] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [CANDIDATE]: Term 1 election: Requested vote from peers 1c00518a0a884679a6116680ff6de161 (127.0.98.2:43845), b05144c5f4cb4e2bae2b21505763849a (127.0.98.1:37547)
I20260619 20:53:44.499749 17717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8d0327d0440eab8bd4bf8169bd132" candidate_uuid: "55a7dc6d6cae45aaa288589d1ac09042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c00518a0a884679a6116680ff6de161"
I20260619 20:53:44.499778 17583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8d0327d0440eab8bd4bf8169bd132" candidate_uuid: "55a7dc6d6cae45aaa288589d1ac09042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05144c5f4cb4e2bae2b21505763849a"
I20260619 20:53:44.499820 17717 raft_consensus.cc:3037] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:44.499832 17583 raft_consensus.cc:3037] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:44.500499 17717 raft_consensus.cc:2454] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a7dc6d6cae45aaa288589d1ac09042 in term 1.
I20260619 20:53:44.500506 17583 raft_consensus.cc:2454] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a7dc6d6cae45aaa288589d1ac09042 in term 1.
I20260619 20:53:44.500702 17785 leader_election.cc:302] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a7dc6d6cae45aaa288589d1ac09042, b05144c5f4cb4e2bae2b21505763849a; no voters:
I20260619 20:53:44.500787 17918 raft_consensus.cc:2781] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:44.500926 17918 raft_consensus.cc:686] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 1 LEADER]: Becoming Leader. State: Replica: 55a7dc6d6cae45aaa288589d1ac09042, State: Running, Role: LEADER
I20260619 20:53:44.501024 17918 consensus_queue.cc:237] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } }
I20260619 20:53:44.501832 17437 catalog_manager.cc:5495] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a7dc6d6cae45aaa288589d1ac09042 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "55a7dc6d6cae45aaa288589d1ac09042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a7dc6d6cae45aaa288589d1ac09042" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 } health_report { overall_health: UNKNOWN } } }
W20260619 20:53:44.520466 17764 tablet.cc:2369] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:53:44.521245 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.518799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.0.98.3:38949)
0619 20:53:44.518849 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.518851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.518860 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:44.518892 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.518892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.518916 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.519000 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:44.519137 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.519600 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.520383 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.520641 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.520718 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.520896 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:44.521009 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:44.521104 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.521124 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.521143 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:53:44.522895 17583 raft_consensus.cc:1261] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 1 FOLLOWER]: Refusing update from remote peer 55a7dc6d6cae45aaa288589d1ac09042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:44.522913 17717 raft_consensus.cc:1261] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 1 FOLLOWER]: Refusing update from remote peer 55a7dc6d6cae45aaa288589d1ac09042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:44.523290 17921 consensus_queue.cc:1035] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b05144c5f4cb4e2bae2b21505763849a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:44.523422 17918 consensus_queue.cc:1035] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c00518a0a884679a6116680ff6de161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:44.526068 17925 mvcc.cc:205] Tried to move back new op lower bound from 7298672330843136000 to 7298672330756816896. Current Snapshot: MvccSnapshot[applied={T|T < 7298672330843136000}]
I20260619 20:53:44.527812 17437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:44.527920 17437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:44.528644 17437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e133b7a35d044fdc8efb77a15054deff: Sending DeleteTablet for 3 replicas of tablet 39e8d0327d0440eab8bd4bf8169bd132
I20260619 20:53:44.528937 17831 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e8d0327d0440eab8bd4bf8169bd132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45338
I20260619 20:53:44.528975 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e8d0327d0440eab8bd4bf8169bd132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40006
I20260619 20:53:44.529031 17563 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e8d0327d0440eab8bd4bf8169bd132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36462
I20260619 20:53:44.529345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17500
I20260619 20:53:44.529399 17935 tablet_replica.cc:330] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a: stopping tablet replica
I20260619 20:53:44.529520 17935 raft_consensus.cc:2229] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:44.529610 17935 raft_consensus.cc:2258] T 39e8d0327d0440eab8bd4bf8169bd132 P b05144c5f4cb4e2bae2b21505763849a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:44.530133 17934 tablet_replica.cc:330] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: stopping tablet replica
I20260619 20:53:44.530242 17934 raft_consensus.cc:2229] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:44.530333 17934 raft_consensus.cc:2258] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:44.531143 17933 tablet_replica.cc:330] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: stopping tablet replica
I20260619 20:53:44.531244 17933 raft_consensus.cc:2229] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:53:44.531330 17933 raft_consensus.cc:2258] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:44.531688 17934 ts_tablet_manager.cc:1905] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:44.533138 17934 ts_tablet_manager.cc:1918] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:44.533144 17933 ts_tablet_manager.cc:1905] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:44.533273 17934 log.cc:1196] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/39e8d0327d0440eab8bd4bf8169bd132
I20260619 20:53:44.533658 17934 ts_tablet_manager.cc:1939] T 39e8d0327d0440eab8bd4bf8169bd132 P 1c00518a0a884679a6116680ff6de161: Deleting consensus metadata
I20260619 20:53:44.534147 17425 catalog_manager.cc:4841] TS 1c00518a0a884679a6116680ff6de161 (127.0.98.2:43845): tablet 39e8d0327d0440eab8bd4bf8169bd132 (table test-table [id=0d8c3203e5414189b4ca92ff9641ecf0]) successfully deleted
I20260619 20:53:44.534742 17933 ts_tablet_manager.cc:1918] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:44.534828 17933 log.cc:1196] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/39e8d0327d0440eab8bd4bf8169bd132
I20260619 20:53:44.535104 17933 ts_tablet_manager.cc:1939] T 39e8d0327d0440eab8bd4bf8169bd132 P 55a7dc6d6cae45aaa288589d1ac09042: Deleting consensus metadata
I20260619 20:53:44.535465 17423 catalog_manager.cc:4841] TS 55a7dc6d6cae45aaa288589d1ac09042 (127.0.98.3:38949): tablet 39e8d0327d0440eab8bd4bf8169bd132 (table test-table [id=0d8c3203e5414189b4ca92ff9641ecf0]) successfully deleted
W20260619 20:53:44.536499 17423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37547 (error 108)
I20260619 20:53:44.536756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17634
I20260619 20:53:44.537341 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.536820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37547 (local address 127.0.0.1:37114)
0619 20:53:44.537237 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.537271 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399}
W20260619 20:53:44.537601 17423 catalog_manager.cc:4584] TS b05144c5f4cb4e2bae2b21505763849a (127.0.98.1:37547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39e8d0327d0440eab8bd4bf8169bd132: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37547: connect: Connection refused (error 111)
I20260619 20:53:44.544066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17768
I20260619 20:53:44.549052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17406
2026-06-19T20:53:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1889 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T20:53:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:44Z Disabled control of system clock
I20260619 20:53:44.575641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44807
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:44.655135 17940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:44.655341 17940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:44.655361 17940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:44.656416 17940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:44.656458 17940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:44.656471 17940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:44.656483 17940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:44.656495 17940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:44.657794 17940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:44.657990 17940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:44.658215 17940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:44.662889 17945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.662900 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.662972 17940 server_base.cc:1034] running on GCE node
W20260619 20:53:44.662905 17946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.663375 17940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:44.663679 17940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:44.664855 17940 hybrid_clock.cc:648] HybridClock initialized: now 1781902424664838 us; error 30 us; skew 500 ppm
I20260619 20:53:44.666167 17940 webserver.cc:460] Webserver started at http://127.0.98.62:36909/ using document root <none> and password file <none>
I20260619 20:53:44.666430 17940 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:44.666504 17940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:44.666642 17940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:44.667665 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "d70fc731e2324d5c8014a6d9c933b297"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "a4696eaaab81b133b459e68939119618"
server_key_iv: "02bced655ba251818bece763342bfd4e"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.668051 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "d70fc731e2324d5c8014a6d9c933b297"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "a4696eaaab81b133b459e68939119618"
server_key_iv: "02bced655ba251818bece763342bfd4e"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.669273 17940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:44.670010 17954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.670162 17940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:44.670258 17940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "d70fc731e2324d5c8014a6d9c933b297"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "a4696eaaab81b133b459e68939119618"
server_key_iv: "02bced655ba251818bece763342bfd4e"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.670339 17940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:44.680353 17940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:44.680639 17940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:44.680794 17940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:44.684640 17940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44807
I20260619 20:53:44.684669 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44807 every 8 connection(s)
I20260619 20:53:44.685045 17940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:44.686029 18007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:44.688212 18007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297: Bootstrap starting.
I20260619 20:53:44.688766 18007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:44.689046 18007 log.cc:824] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:44.689567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 17940
I20260619 20:53:44.689690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:44.689869 18007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297: No bootstrap required, opened a new log
I20260619 20:53:44.689889 392 external_mini_cluster.cc:1428] Setting key 8e43448081ab9b199e73cca3133bbc32
I20260619 20:53:44.692802 18007 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fc731e2324d5c8014a6d9c933b297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44807 } }
I20260619 20:53:44.692970 18007 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:44.692999 18007 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70fc731e2324d5c8014a6d9c933b297, State: Initialized, Role: FOLLOWER
I20260619 20:53:44.693110 18007 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fc731e2324d5c8014a6d9c933b297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44807 } }
I20260619 20:53:44.693181 18007 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:44.693215 18007 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:44.693269 18007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:44.693866 18007 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fc731e2324d5c8014a6d9c933b297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44807 } }
I20260619 20:53:44.693954 18007 leader_election.cc:302] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70fc731e2324d5c8014a6d9c933b297; no voters:
I20260619 20:53:44.694151 18007 leader_election.cc:288] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:44.694178 18012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:44.694293 18012 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [term 1 LEADER]: Becoming Leader. State: Replica: d70fc731e2324d5c8014a6d9c933b297, State: Running, Role: LEADER
I20260619 20:53:44.694414 18007 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:44.694429 18012 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fc731e2324d5c8014a6d9c933b297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44807 } }
I20260619 20:53:44.694854 18014 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70fc731e2324d5c8014a6d9c933b297. Latest consensus state: current_term: 1 leader_uuid: "d70fc731e2324d5c8014a6d9c933b297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fc731e2324d5c8014a6d9c933b297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44807 } } }
I20260619 20:53:44.695119 18013 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70fc731e2324d5c8014a6d9c933b297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fc731e2324d5c8014a6d9c933b297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44807 } } }
I20260619 20:53:44.695201 18013 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:44.695029 18014 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:44.695449 18021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:44.695978 18021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:44.696441 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.690451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.0.98.62:44807)
0619 20:53:44.690661 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.690669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.690700 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:53:44.691270 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.691279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.691368 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.691524 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.691735 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.693168 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.694105 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.694709 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.694867 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.695579 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.695583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.695583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:44.695784 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.695824 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:44.695836 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.695894 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.696167 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.696243 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260619 20:53:44.697674 18021 catalog_manager.cc:1347] Generated new cluster ID: a9f641b8d7384958ac7bde809db8e02e
I20260619 20:53:44.697727 18021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:44.711467 18021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:44.712111 18021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:44.714679 18021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297: Generated new TSK 0
I20260619 20:53:44.714855 18021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:44.721488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44807
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:44.808591 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:44.808791 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:44.808813 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:44.810032 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:44.810070 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:44.810096 18031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:44.811424 18031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:44.811648 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:44.811957 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:44.817662 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.817678 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.817672 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.817765 18031 server_base.cc:1034] running on GCE node
I20260619 20:53:44.818157 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:44.818393 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:44.819557 18031 hybrid_clock.cc:648] HybridClock initialized: now 1781902424819543 us; error 30 us; skew 500 ppm
I20260619 20:53:44.820883 18031 webserver.cc:460] Webserver started at http://127.0.98.1:44149/ using document root <none> and password file <none>
I20260619 20:53:44.821120 18031 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:44.821188 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:44.821357 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:44.822430 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "311f24bcb1e5433181f620f16aa15c6d"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "b9f0770f9acf70310eccdea0079a7069"
server_key_iv: "d9c07112a13ee22bad6e0d90ee8acded"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.822791 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "311f24bcb1e5433181f620f16aa15c6d"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "b9f0770f9acf70310eccdea0079a7069"
server_key_iv: "d9c07112a13ee22bad6e0d90ee8acded"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.824105 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:53:44.824859 18045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.825049 18031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:44.825115 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "311f24bcb1e5433181f620f16aa15c6d"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "b9f0770f9acf70310eccdea0079a7069"
server_key_iv: "d9c07112a13ee22bad6e0d90ee8acded"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.825204 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:44.837708 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:44.837972 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:44.838075 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:44.838264 18031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:44.838526 18031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:44.838557 18031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.838579 18031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:44.838593 18031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.844417 18031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37003
I20260619 20:53:44.844463 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37003 every 8 connection(s)
I20260619 20:53:44.844779 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:44.846735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18031
I20260619 20:53:44.846870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:53:44.847067 392 external_mini_cluster.cc:1428] Setting key 93da5d25b0e55a1b24e6f48a2db05a43
I20260619 20:53:44.848039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44807
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:53:44.850561 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.845524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39257 (local address 127.0.98.62:44807)
0619 20:53:44.845584 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.845587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.845816 (+ 229us) server_negotiation.cc:407] Connection header received
0619 20:53:44.846353 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.846354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.846370 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.846435 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.847176 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.847845 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.848888 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.849231 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.849333 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.850241 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.850242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.850242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:44.850247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.850265 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:44.850269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.850329 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.850416 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.850442 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:53:44.850680 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.845514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44807 (local address 127.0.98.1:39257)
0619 20:53:44.845694 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.845732 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.846247 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.846503 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.846507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.846516 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:44.847083 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.847088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.847931 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.847932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.848800 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.848802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.849359 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.849360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.849491 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.850070 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:44.850071 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:44.850100 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:44.850123 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:44.850339 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:44.850343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:44.850346 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.850399 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.850480 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260619 20:53:44.851152 18159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44807
I20260619 20:53:44.851266 18159 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:44.851450 18159 heartbeater.cc:511] Master 127.0.98.62:44807 requested a full tablet report, sending...
I20260619 20:53:44.852001 17971 ts_manager.cc:194] Registered new tserver with Master: 311f24bcb1e5433181f620f16aa15c6d (127.0.98.1:37003)
I20260619 20:53:44.852993 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39257
W20260619 20:53:44.928448 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:44.928758 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:44.928819 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:44.929970 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:44.930030 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:44.930074 18162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:44.931514 18162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:44.931782 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:44.932078 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:44.937004 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:44.937022 18170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.937040 18162 server_base.cc:1034] running on GCE node
W20260619 20:53:44.937047 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:44.937525 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:44.937796 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:44.938962 18162 hybrid_clock.cc:648] HybridClock initialized: now 1781902424938949 us; error 26 us; skew 500 ppm
I20260619 20:53:44.940285 18162 webserver.cc:460] Webserver started at http://127.0.98.2:45905/ using document root <none> and password file <none>
I20260619 20:53:44.940536 18162 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:44.940618 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:44.940785 18162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:44.941927 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "316496e8212f470da86fb3922c9955cb"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "6296c0c7df21a62873ad84ac6f580286"
server_key_iv: "89139924aa0748638b74e03254e754aa"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.942302 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "316496e8212f470da86fb3922c9955cb"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "6296c0c7df21a62873ad84ac6f580286"
server_key_iv: "89139924aa0748638b74e03254e754aa"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.943609 18162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:53:44.944296 18176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.944458 18162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:44.944563 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "316496e8212f470da86fb3922c9955cb"
format_stamp: "Formatted at 2026-06-19 20:53:44 on dist-test-slave-0smx"
server_key: "6296c0c7df21a62873ad84ac6f580286"
server_key_iv: "89139924aa0748638b74e03254e754aa"
server_key_version: "encryptionkey@0"
I20260619 20:53:44.944660 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:44.960623 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:44.960933 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:44.961092 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:44.961371 18162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:44.961717 18162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:44.961782 18162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.961826 18162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:44.961855 18162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:44.968216 18162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35495
I20260619 20:53:44.968267 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35495 every 8 connection(s)
I20260619 20:53:44.968657 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:44.973004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18162
I20260619 20:53:44.973110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:53:44.973366 392 external_mini_cluster.cc:1428] Setting key 48bceaedf50b8c025987ae86457228ac
I20260619 20:53:44.974182 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.969392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47207 (local address 127.0.98.62:44807)
0619 20:53:44.969450 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:44.969452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:44.969695 (+ 243us) server_negotiation.cc:407] Connection header received
0619 20:53:44.970174 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:44.970175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:44.970185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:44.970233 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:44.970873 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.971363 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.972438 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.972722 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.972800 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.973922 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:44.973924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:44.973924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:44.973929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:44.973944 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:44.973948 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:44.973998 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:44.974076 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:53:44.974104 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:53:44.974274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44807
--builtin_ntp_servers=127.0.98.20:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:53:44.974413 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:44.969380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44807 (local address 127.0.98.2:47207)
0619 20:53:44.969567 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:44.969603 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:44.970082 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:44.970297 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:44.970300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:44.970307 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:44.970804 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.970808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.971495 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.971496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.972329 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:44.972331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:44.972850 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:44.972851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:44.973023 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:44.973759 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:44.973760 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:44.973791 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:44.973814 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:44.974003 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:44.974008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:44.974011 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:44.974080 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:53:44.974184 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260619 20:53:44.974843 18290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44807
I20260619 20:53:44.974934 18290 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:44.975133 18290 heartbeater.cc:511] Master 127.0.98.62:44807 requested a full tablet report, sending...
I20260619 20:53:44.975502 17971 ts_manager.cc:194] Registered new tserver with Master: 316496e8212f470da86fb3922c9955cb (127.0.98.2:35495)
I20260619 20:53:44.975875 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47207
W20260619 20:53:45.056048 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:45.056262 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:45.056283 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:45.057356 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:45.057389 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:45.057413 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:45.058815 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:45.059270 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:45.059659 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:45.064507 18298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:45.064612 18293 server_base.cc:1034] running on GCE node
W20260619 20:53:45.064530 18301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:45.064543 18299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:45.064927 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:45.065166 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:45.066332 18293 hybrid_clock.cc:648] HybridClock initialized: now 1781902425066320 us; error 27 us; skew 500 ppm
I20260619 20:53:45.067504 18293 webserver.cc:460] Webserver started at http://127.0.98.3:33057/ using document root <none> and password file <none>
I20260619 20:53:45.067737 18293 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:45.067811 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:45.067945 18293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:45.068985 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "f048e9ff17ea45d7b1637ce40976c4ce"
format_stamp: "Formatted at 2026-06-19 20:53:45 on dist-test-slave-0smx"
server_key: "3989be59bbb79f2f8806de480e01fe3d"
server_key_iv: "7453b729e483fbc995e61be3d4ac68cc"
server_key_version: "encryptionkey@0"
I20260619 20:53:45.069401 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f048e9ff17ea45d7b1637ce40976c4ce"
format_stamp: "Formatted at 2026-06-19 20:53:45 on dist-test-slave-0smx"
server_key: "3989be59bbb79f2f8806de480e01fe3d"
server_key_iv: "7453b729e483fbc995e61be3d4ac68cc"
server_key_version: "encryptionkey@0"
I20260619 20:53:45.070653 18293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:45.071370 18307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:45.071553 18293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:53:45.071664 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "f048e9ff17ea45d7b1637ce40976c4ce"
format_stamp: "Formatted at 2026-06-19 20:53:45 on dist-test-slave-0smx"
server_key: "3989be59bbb79f2f8806de480e01fe3d"
server_key_iv: "7453b729e483fbc995e61be3d4ac68cc"
server_key_version: "encryptionkey@0"
I20260619 20:53:45.071749 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:45.088411 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:45.088688 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:45.088840 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:45.089056 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:45.089425 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:45.089479 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:45.089519 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:45.089546 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:45.095402 18293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35015
I20260619 20:53:45.095451 18420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35015 every 8 connection(s)
I20260619 20:53:45.095795 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:53:45.099190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18293
I20260619 20:53:45.099272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:53:45.099467 392 external_mini_cluster.cc:1428] Setting key 13a39473919db505a22cf462242bd417
I20260619 20:53:45.101193 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.096552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35715 (local address 127.0.98.62:44807)
0619 20:53:45.096617 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.096619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.096853 (+ 234us) server_negotiation.cc:407] Connection header received
0619 20:53:45.097412 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.097413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.097427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.097488 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:45.098172 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.098680 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.099916 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.100134 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.100212 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.100941 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:45.100942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:45.100943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:45.100947 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:45.100961 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:45.100964 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:45.101004 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.101086 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.101109 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:53:45.101426 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.096536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44807 (local address 127.0.98.3:35715)
0619 20:53:45.096737 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.096768 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:45.097271 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:45.097549 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:45.097552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:45.097562 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:45.098072 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.098076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.098777 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.098778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.099836 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.099839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.100230 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.100231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.100367 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.100801 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:45.100801 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:45.100827 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:45.100848 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:45.101016 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:45.101020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:45.101022 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:45.101079 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:53:45.101172 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260619 20:53:45.101787 18421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44807
I20260619 20:53:45.101858 18421 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:45.102041 18421 heartbeater.cc:511] Master 127.0.98.62:44807 requested a full tablet report, sending...
I20260619 20:53:45.102365 17971 ts_manager.cc:194] Registered new tserver with Master: f048e9ff17ea45d7b1637ce40976c4ce (127.0.98.3:35015)
I20260619 20:53:45.102723 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35715
I20260619 20:53:45.111044 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:45.115128 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.112369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.62:44807)
0619 20:53:45.112425 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.112427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.112553 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:53:45.112575 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.112576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.112584 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.112634 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:45.112771 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.113245 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.114020 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.114218 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.114287 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.114879 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:45.114881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:45.114881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:45.114885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:45.114898 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:45.114902 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:45.114947 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.114992 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.115011 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:53:45.116526 17971 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:45.118163 17971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:45.124179 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.120238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.0.1:47442)
0619 20:53:45.120409 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.120447 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:45.120509 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:45.120778 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:45.120782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:45.120785 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:45.120854 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.120855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.121932 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.121933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.123152 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.123155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.123879 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.123879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.123954 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.123965 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 20:53:45.124055 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 20:53:45.124106 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260619 20:53:45.124197 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.120462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.0.98.3:35015)
0619 20:53:45.120621 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.120626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.120638 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:53:45.120680 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.120682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.120698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.120778 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:45.120952 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.121849 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.123231 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.123790 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.123877 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.123967 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.124069 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.124106 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260619 20:53:45.124584 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.120253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.0.1:48652)
0619 20:53:45.120615 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.120625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:45.120674 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:45.120861 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:45.120862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:45.120864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:45.120912 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.120914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.122235 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.122236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.123276 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.123278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.124307 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.124308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.124371 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.124377 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:53:45.124439 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:53:45.124489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":7,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260619 20:53:45.124604 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.120327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.0.98.2:35495)
0619 20:53:45.120686 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.120691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.120701 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:53:45.120743 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.120744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.120761 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.120843 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:45.121004 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.122150 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.123371 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.124112 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.124191 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.124268 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.124468 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.124510 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":86,"threads_started":1}
I20260619 20:53:45.124852 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.120483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.0.98.1:37003)
0619 20:53:45.120897 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.120902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.120911 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:45.120948 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.120950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.120966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.121037 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:45.121226 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.122324 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.123424 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.124112 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.124185 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.124268 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.124723 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.124772 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":73,"threads_started":1}
I20260619 20:53:45.124584 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.120255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37003 (local address 127.0.0.1:39622)
0619 20:53:45.120627 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.120638 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:45.120699 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:45.121104 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:45.121105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:45.121107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:45.121136 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.121137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.122405 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.122406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.123343 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.123346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.124306 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.124307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.124372 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.124378 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:53:45.124437 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:53:45.124488 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260619 20:53:45.125344 18355 tablet_service.cc:1467] Processing CreateTablet for tablet 6277e78f9e9944cba43f0930ff96cce0 (DEFAULT_TABLE table=test-table [id=861a6e6a41574f359d9c18a1f9b7f298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:45.125586 18224 tablet_service.cc:1467] Processing CreateTablet for tablet 6277e78f9e9944cba43f0930ff96cce0 (DEFAULT_TABLE table=test-table [id=861a6e6a41574f359d9c18a1f9b7f298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:45.125747 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277e78f9e9944cba43f0930ff96cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:45.125885 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277e78f9e9944cba43f0930ff96cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:45.128392 18435 tablet_bootstrap.cc:492] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: Bootstrap starting.
I20260619 20:53:45.128722 18093 tablet_service.cc:1467] Processing CreateTablet for tablet 6277e78f9e9944cba43f0930ff96cce0 (DEFAULT_TABLE table=test-table [id=861a6e6a41574f359d9c18a1f9b7f298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:45.129020 18435 tablet_bootstrap.cc:654] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:45.129020 18093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277e78f9e9944cba43f0930ff96cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:45.129374 18435 log.cc:824] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:45.129462 18436 tablet_bootstrap.cc:492] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Bootstrap starting.
I20260619 20:53:45.130059 18436 tablet_bootstrap.cc:654] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:45.130206 18435 tablet_bootstrap.cc:492] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: No bootstrap required, opened a new log
I20260619 20:53:45.130280 18435 ts_tablet_manager.cc:1397] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:45.130376 18436 log.cc:824] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:45.131184 18436 tablet_bootstrap.cc:492] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: No bootstrap required, opened a new log
I20260619 20:53:45.131266 18436 ts_tablet_manager.cc:1397] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:45.132251 18439 tablet_bootstrap.cc:492] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d: Bootstrap starting.
I20260619 20:53:45.132824 18439 tablet_bootstrap.cc:654] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:45.133105 18439 log.cc:824] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:45.133044 18435 raft_consensus.cc:348] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.133250 18435 raft_consensus.cc:374] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:45.133340 18435 raft_consensus.cc:729] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316496e8212f470da86fb3922c9955cb, State: Initialized, Role: FOLLOWER
I20260619 20:53:45.133499 18435 consensus_queue.cc:260] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.133770 18435 ts_tablet_manager.cc:1428] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 20:53:45.133852 18439 tablet_bootstrap.cc:492] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d: No bootstrap required, opened a new log
I20260619 20:53:45.133891 18290 heartbeater.cc:503] Master 127.0.98.62:44807 was elected leader, sending a full tablet report...
I20260619 20:53:45.133920 18439 ts_tablet_manager.cc:1397] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:45.133994 18436 raft_consensus.cc:348] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.134183 18436 raft_consensus.cc:374] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:45.134233 18436 raft_consensus.cc:729] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f048e9ff17ea45d7b1637ce40976c4ce, State: Initialized, Role: FOLLOWER
I20260619 20:53:45.134341 18436 consensus_queue.cc:260] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.134617 18436 ts_tablet_manager.cc:1428] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:53:45.134634 18421 heartbeater.cc:503] Master 127.0.98.62:44807 was elected leader, sending a full tablet report...
I20260619 20:53:45.136240 18439 raft_consensus.cc:348] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.136358 18439 raft_consensus.cc:374] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:45.136380 18439 raft_consensus.cc:729] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311f24bcb1e5433181f620f16aa15c6d, State: Initialized, Role: FOLLOWER
I20260619 20:53:45.136513 18439 consensus_queue.cc:260] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.136737 18439 ts_tablet_manager.cc:1428] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:53:45.136824 18159 heartbeater.cc:503] Master 127.0.98.62:44807 was elected leader, sending a full tablet report...
I20260619 20:53:45.181944 18441 raft_consensus.cc:482] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:45.182067 18441 raft_consensus.cc:504] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.182397 18441 leader_election.cc:288] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f048e9ff17ea45d7b1637ce40976c4ce (127.0.98.3:35015), 311f24bcb1e5433181f620f16aa15c6d (127.0.98.1:37003)
I20260619 20:53:45.185093 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.182540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37003 (local address 127.0.98.2:50785)
0619 20:53:45.182598 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.182609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:45.182647 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:45.182805 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:45.182806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:45.182807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:45.182887 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.182888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.183519 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.183520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.184262 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.184264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.184824 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.184825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.184900 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.184902 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:45.184977 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:53:45.185018 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:53:45.185093 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.182618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50785 (local address 127.0.98.1:37003)
0619 20:53:45.182677 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.182679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.182687 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:45.182714 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.182715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.182725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.182800 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:45.182962 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.183435 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.184377 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.184741 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.184823 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.184842 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.184987 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.185027 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:53:45.185246 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.182557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.98.2:54923)
0619 20:53:45.182721 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.182731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:45.182825 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:45.182968 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:45.182969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:45.182970 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:45.183043 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.183044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.183663 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.183664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.184408 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:45.184410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.184963 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.184964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:45.185062 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.185065 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:45.185125 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:53:45.185164 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260619 20:53:45.185240 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.182605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54923 (local address 127.0.98.3:35015)
0619 20:53:45.182654 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.182656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.182811 (+ 155us) server_negotiation.cc:407] Connection header received
0619 20:53:45.182886 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.182887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.182896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.182966 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:45.183121 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.183582 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.184507 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.184868 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.184959 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.184967 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.185143 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.185182 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:53:45.185537 18113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277e78f9e9944cba43f0930ff96cce0" candidate_uuid: "316496e8212f470da86fb3922c9955cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f24bcb1e5433181f620f16aa15c6d" is_pre_election: true
I20260619 20:53:45.185607 18375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277e78f9e9944cba43f0930ff96cce0" candidate_uuid: "316496e8212f470da86fb3922c9955cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" is_pre_election: true
I20260619 20:53:45.185706 18113 raft_consensus.cc:2454] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316496e8212f470da86fb3922c9955cb in term 0.
I20260619 20:53:45.185720 18375 raft_consensus.cc:2454] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316496e8212f470da86fb3922c9955cb in term 0.
I20260619 20:53:45.185935 18177 leader_election.cc:302] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316496e8212f470da86fb3922c9955cb, f048e9ff17ea45d7b1637ce40976c4ce; no voters:
I20260619 20:53:45.186080 18441 raft_consensus.cc:2781] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:45.186137 18441 raft_consensus.cc:482] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:45.186162 18441 raft_consensus.cc:3037] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:45.186856 18441 raft_consensus.cc:504] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.186998 18441 leader_election.cc:288] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [CANDIDATE]: Term 1 election: Requested vote from peers f048e9ff17ea45d7b1637ce40976c4ce (127.0.98.3:35015), 311f24bcb1e5433181f620f16aa15c6d (127.0.98.1:37003)
I20260619 20:53:45.187147 18375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277e78f9e9944cba43f0930ff96cce0" candidate_uuid: "316496e8212f470da86fb3922c9955cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f048e9ff17ea45d7b1637ce40976c4ce"
I20260619 20:53:45.187170 18113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277e78f9e9944cba43f0930ff96cce0" candidate_uuid: "316496e8212f470da86fb3922c9955cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311f24bcb1e5433181f620f16aa15c6d"
I20260619 20:53:45.187229 18375 raft_consensus.cc:3037] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:45.187233 18113 raft_consensus.cc:3037] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:45.188124 18113 raft_consensus.cc:2454] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316496e8212f470da86fb3922c9955cb in term 1.
I20260619 20:53:45.188124 18375 raft_consensus.cc:2454] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316496e8212f470da86fb3922c9955cb in term 1.
I20260619 20:53:45.188324 18177 leader_election.cc:302] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316496e8212f470da86fb3922c9955cb, f048e9ff17ea45d7b1637ce40976c4ce; no voters:
I20260619 20:53:45.188442 18441 raft_consensus.cc:2781] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:45.188618 18441 raft_consensus.cc:686] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 1 LEADER]: Becoming Leader. State: Replica: 316496e8212f470da86fb3922c9955cb, State: Running, Role: LEADER
I20260619 20:53:45.188719 18441 consensus_queue.cc:237] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } }
I20260619 20:53:45.189658 17970 catalog_manager.cc:5495] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 316496e8212f470da86fb3922c9955cb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "316496e8212f470da86fb3922c9955cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316496e8212f470da86fb3922c9955cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 } health_report { overall_health: UNKNOWN } } }
I20260619 20:53:45.196552 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.194059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.0.98.2:35495)
0619 20:53:45.194111 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:45.194113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:45.194120 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:53:45.194191 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:45.194192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:45.194232 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:45.194297 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:45.194381 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.194964 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.195758 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:45.195982 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:45.196034 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:45.196247 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:45.196361 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:45.196442 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:45.196476 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 20:53:45.196491 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:53:45.198096 18113 raft_consensus.cc:1261] T 6277e78f9e9944cba43f0930ff96cce0 P 311f24bcb1e5433181f620f16aa15c6d [term 1 FOLLOWER]: Refusing update from remote peer 316496e8212f470da86fb3922c9955cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:45.198122 18375 raft_consensus.cc:1261] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 1 FOLLOWER]: Refusing update from remote peer 316496e8212f470da86fb3922c9955cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:45.198469 18441 consensus_queue.cc:1035] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "311f24bcb1e5433181f620f16aa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:45.198601 18445 consensus_queue.cc:1035] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f048e9ff17ea45d7b1637ce40976c4ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:45.203059 17970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:45.203145 17970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:45.203884 17970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d70fc731e2324d5c8014a6d9c933b297: Sending DeleteTablet for 3 replicas of tablet 6277e78f9e9944cba43f0930ff96cce0
I20260619 20:53:45.204208 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet 6277e78f9e9944cba43f0930ff96cce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:45 UTC) from {username='slave'} at 127.0.0.1:47442
I20260619 20:53:45.204249 18093 tablet_service.cc:1514] Processing DeleteTablet for tablet 6277e78f9e9944cba43f0930ff96cce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:45 UTC) from {username='slave'} at 127.0.0.1:39622
I20260619 20:53:45.204301 18224 tablet_service.cc:1514] Processing DeleteTablet for tablet 6277e78f9e9944cba43f0930ff96cce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:45 UTC) from {username='slave'} at 127.0.0.1:48652
I20260619 20:53:45.204444 18458 tablet_replica.cc:330] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: stopping tablet replica
I20260619 20:53:45.204543 18458 raft_consensus.cc:2229] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:45.204636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18031
I20260619 20:53:45.204843 18458 raft_consensus.cc:2258] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:45.205489 18458 ts_tablet_manager.cc:1905] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:45.207361 18458 ts_tablet_manager.cc:1918] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:45.207437 18458 log.cc:1196] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/6277e78f9e9944cba43f0930ff96cce0
I20260619 20:53:45.207682 18458 ts_tablet_manager.cc:1939] T 6277e78f9e9944cba43f0930ff96cce0 P f048e9ff17ea45d7b1637ce40976c4ce: Deleting consensus metadata
I20260619 20:53:45.208063 17955 catalog_manager.cc:4841] TS f048e9ff17ea45d7b1637ce40976c4ce (127.0.98.3:35015): tablet 6277e78f9e9944cba43f0930ff96cce0 (table test-table [id=861a6e6a41574f359d9c18a1f9b7f298]) successfully deleted
W20260619 20:53:45.212267 17956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37003 (error 108)
I20260619 20:53:45.212337 18460 tablet_replica.cc:330] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: stopping tablet replica
I20260619 20:53:45.212495 18460 raft_consensus.cc:2229] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:53:45.212599 18460 raft_consensus.cc:2258] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:45.212991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18162
I20260619 20:53:45.213021 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.212881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37003 (local address 127.0.0.1:39628)
0619 20:53:45.212942 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.212988 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:53:45.213088 18460 ts_tablet_manager.cc:1905] T 6277e78f9e9944cba43f0930ff96cce0 P 316496e8212f470da86fb3922c9955cb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:53:45.214645 17956 catalog_manager.cc:4584] TS 311f24bcb1e5433181f620f16aa15c6d (127.0.98.1:37003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6277e78f9e9944cba43f0930ff96cce0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37003: connect: Connection refused (error 111)
I20260619 20:53:45.217408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18293
I20260619 20:53:45.217489 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:45.217365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.0.1:48672)
0619 20:53:45.217414 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:45.217442 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:53:45.221686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 17940
2026-06-19T20:53:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (672 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:45 dist-test-slave-0smx krb5kdc[18464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:45 dist-test-slave-0smx krb5kdc[18464](info): set up 2 sockets
Jun 19 20:53:45 dist-test-slave-0smx krb5kdc[18464](info): commencing operation
krb5kdc: starting...
W20260619 20:53:46.261171 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:53:46.374141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37117
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:46.455118 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:46.455322 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:46.455343 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:46.456399 18477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:46.456434 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:46.456449 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:46.456461 18477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:46.456473 18477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:46.457907 18477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:46.458142 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:46.458334 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:53:46.463135 18477 server_base.cc:1034] running on GCE node
W20260619 20:53:46.463119 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:46.463117 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:46.463105 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:46.463559 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:46.463851 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:46.465010 18477 hybrid_clock.cc:648] HybridClock initialized: now 1781902426464987 us; error 52 us; skew 500 ppm
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:46.467516 18477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:53:46.467985 18477 webserver.cc:460] Webserver started at http://127.0.98.62:38863/ using document root <none> and password file <none>
I20260619 20:53:46.468204 18477 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:46.468286 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:46.468391 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:46.469451 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "3b9ba415a6b0419ba6f57be6d6b0da01"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "8e47c97bd6db878f9a06456082deef36"
server_key_iv: "8652bc806eb65ba658377d112393211a"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.469753 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b9ba415a6b0419ba6f57be6d6b0da01"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "8e47c97bd6db878f9a06456082deef36"
server_key_iv: "8652bc806eb65ba658377d112393211a"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.470980 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:46.471637 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.471815 18477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.471911 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "3b9ba415a6b0419ba6f57be6d6b0da01"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "8e47c97bd6db878f9a06456082deef36"
server_key_iv: "8652bc806eb65ba658377d112393211a"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.471990 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:46.509444 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:46.511755 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:46.511929 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:46.516268 18477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37117
I20260619 20:53:46.516324 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37117 every 8 connection(s)
I20260619 20:53:46.516683 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:46.517588 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:46.518998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18477
I20260619 20:53:46.519106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:46.519296 392 external_mini_cluster.cc:1428] Setting key a46de351fcf1ada5b02c6f4aa8f4c51c
I20260619 20:53:46.520231 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01: Bootstrap starting.
I20260619 20:53:46.520921 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:46.521239 18545 log.cc:824] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:46.521965 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01: No bootstrap required, opened a new log
I20260619 20:53:46.524169 18545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37117 } }
I20260619 20:53:46.524348 18545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:46.524386 18545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9ba415a6b0419ba6f57be6d6b0da01, State: Initialized, Role: FOLLOWER
I20260619 20:53:46.524544 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37117 } }
I20260619 20:53:46.524621 18545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:46.524669 18545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:46.524729 18545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:46.525352 18545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37117 } }
I20260619 20:53:46.525478 18545 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b9ba415a6b0419ba6f57be6d6b0da01; no voters:
I20260619 20:53:46.525636 18545 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:46.525681 18550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:46.525813 18550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9ba415a6b0419ba6f57be6d6b0da01, State: Running, Role: LEADER
I20260619 20:53:46.525928 18545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:46.525985 18550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37117 } }
I20260619 20:53:46.526325 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b9ba415a6b0419ba6f57be6d6b0da01. Latest consensus state: current_term: 1 leader_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37117 } } }
I20260619 20:53:46.526332 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ba415a6b0419ba6f57be6d6b0da01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37117 } } }
I20260619 20:53:46.526495 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [sys.catalog]: This master's current role is: LEADER
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:46.526697 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:46.527099 18558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:46.527699 18558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:46.529687 18558 catalog_manager.cc:1347] Generated new cluster ID: e7c75f56693c402cbc302b8353e6d07a
I20260619 20:53:46.529737 18558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:46.530694 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:46.519810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.0.98.62:37117)
0619 20:53:46.520022 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:46.520031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:46.520063 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:53:46.520618 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:46.520629 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:46.520701 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:46.520857 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:46.521526 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.522857 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.523625 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.524291 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.524408 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:46.527164 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:46.527168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:46.527168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:46.527179 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:46.529066 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:46.529499 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:46.529501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:46.529502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:46.529636 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:46.529863 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:46.529864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:46.529864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:46.530090 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:46.530262 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:46.530350 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 20:53:46.530482 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260619 20:53:46.553721 18558 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:46.554296 18558 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:46.567517 18558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01: Generated new TSK 0
I20260619 20:53:46.567766 18558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:53:46.598028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37117
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:46.678337 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:46.678541 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:46.678565 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:46.679644 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:46.679680 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:46.679718 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:46.681089 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:46.681252 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:46.681430 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:46.686074 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:46.686066 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:46.686082 18571 server_base.cc:1034] running on GCE node
W20260619 20:53:46.686051 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:46.686460 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:46.686729 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:46.687870 18571 hybrid_clock.cc:648] HybridClock initialized: now 1781902426687854 us; error 29 us; skew 500 ppm
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:46.690339 18571 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:53:46.690749 18571 webserver.cc:460] Webserver started at http://127.0.98.1:40655/ using document root <none> and password file <none>
I20260619 20:53:46.690956 18571 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:46.691011 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:46.691169 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:46.692198 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "8957652d881744fca12c828872cb6745"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "42719003cda99d9f26bb455e672b98cf"
server_key_iv: "509551a1d55085c40586ad7b2e56438a"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.692559 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8957652d881744fca12c828872cb6745"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "42719003cda99d9f26bb455e672b98cf"
server_key_iv: "509551a1d55085c40586ad7b2e56438a"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.693902 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:53:46.694576 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.694766 18571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:46.694869 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "8957652d881744fca12c828872cb6745"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "42719003cda99d9f26bb455e672b98cf"
server_key_iv: "509551a1d55085c40586ad7b2e56438a"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.694941 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:46.710487 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:46.712163 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:46.712312 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:46.712551 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:46.712872 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:46.712904 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.712957 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:46.712982 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.718693 18571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45015
I20260619 20:53:46.718783 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45015 every 8 connection(s)
I20260619 20:53:46.719086 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:46.722678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18571
I20260619 20:53:46.722764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:53:46.722962 392 external_mini_cluster.cc:1428] Setting key 685bba29e783b7b50c916f744d01b2e5
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:46.729230 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:46.719851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43147 (local address 127.0.98.62:37117)
0619 20:53:46.719915 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:46.719917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:46.720115 (+ 198us) server_negotiation.cc:407] Connection header received
0619 20:53:46.721018 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:46.721020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:46.721028 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:46.721081 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:46.721688 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.722216 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.723494 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.723801 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.723872 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:46.726151 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:46.726153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:46.726153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:46.726161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:46.727812 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:46.728237 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:46.728238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:46.728238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:46.728296 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:46.728554 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:46.728555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:46.728555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:46.728686 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:46.728759 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:46.729046 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 20:53:46.729138 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:53:46.729458 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:46.719832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37117 (local address 127.0.98.1:43147)
0619 20:53:46.719990 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:46.720029 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:46.720932 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:46.721142 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:46.721145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:46.721418 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:46.721606 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:46.721610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.722294 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.722294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:46.723401 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:46.723405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.723884 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.723884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:46.723989 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:46.724536 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:46.724545 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:46.726047 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:46.727914 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:46.727922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:46.727924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:46.728156 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:46.728371 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:46.728372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:46.728373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:46.728483 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:46.728762 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:46.728769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:46.728898 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 20:53:46.729021 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:53:46.729180 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260619 20:53:46.730012 18700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37117
I20260619 20:53:46.730103 18700 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:46.730293 18700 heartbeater.cc:511] Master 127.0.98.62:37117 requested a full tablet report, sending...
I20260619 20:53:46.730896 18509 ts_manager.cc:194] Registered new tserver with Master: 8957652d881744fca12c828872cb6745 (127.0.98.1:45015)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 20:53:46.731817 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:53:46.747954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37117
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:46.825860 18705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:46.826071 18705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:46.826094 18705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:46.827317 18705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:46.827356 18705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:46.827382 18705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:46.828783 18705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:46.828958 18705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:46.829139 18705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:53:46.834110 18705 server_base.cc:1034] running on GCE node
W20260619 20:53:46.834236 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:46.834292 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:46.834403 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:46.834664 18705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:46.834905 18705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:46.836043 18705 hybrid_clock.cc:648] HybridClock initialized: now 1781902426836029 us; error 32 us; skew 500 ppm
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:46.838457 18705 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:53:46.838887 18705 webserver.cc:460] Webserver started at http://127.0.98.2:44667/ using document root <none> and password file <none>
I20260619 20:53:46.839107 18705 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:46.839174 18705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:46.839329 18705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:46.840312 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "76990b59d57546d1b3c44e6004bd9479"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "60552ca1de4c146b577a82987976229a"
server_key_iv: "4fa78f309471b8431d4dae71738e2ab8"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.840667 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76990b59d57546d1b3c44e6004bd9479"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "60552ca1de4c146b577a82987976229a"
server_key_iv: "4fa78f309471b8431d4dae71738e2ab8"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.842088 18705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:46.842759 18720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.842923 18705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:46.843017 18705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "76990b59d57546d1b3c44e6004bd9479"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "60552ca1de4c146b577a82987976229a"
server_key_iv: "4fa78f309471b8431d4dae71738e2ab8"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.843093 18705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:46.856256 18705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:46.858080 18705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:46.858227 18705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:46.858429 18705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:46.858745 18705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:46.858799 18705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.858839 18705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:46.858865 18705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.864687 18705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42037
I20260619 20:53:46.864728 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42037 every 8 connection(s)
I20260619 20:53:46.865024 18705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:46.872421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18705
I20260619 20:53:46.872526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:53:46.872713 392 external_mini_cluster.cc:1428] Setting key 4a7f068bf4663e417d50a8b2535c08b0
I20260619 20:53:46.875308 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:46.865669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37117 (local address 127.0.98.62:37117)
0619 20:53:46.865719 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:46.865720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:46.865959 (+ 239us) server_negotiation.cc:407] Connection header received
0619 20:53:46.866940 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:46.866941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:46.866952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:46.867016 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:46.867654 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.868201 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.869207 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.869506 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.869574 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:46.872019 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:46.872021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:46.872021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:46.872028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:46.873720 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:46.874206 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:46.874207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:46.874207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:46.874260 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:46.874568 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:46.874569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:46.874569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:46.874671 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:46.874726 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:46.875092 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 20:53:46.875185 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:53:46.875414 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:46.865637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37117 (local address 127.0.98.2:37117)
0619 20:53:46.865816 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:46.865858 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:46.866817 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:46.867105 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:46.867108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:46.867349 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:46.867539 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:46.867543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.868285 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.868285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:46.869107 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:46.869109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:46.869596 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:46.869597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:46.869703 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:46.870205 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:46.870214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:46.871924 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:46.873795 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:46.873800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:46.873803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:46.874123 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:46.874330 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:46.874331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:46.874332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:46.874477 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:46.874737 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:46.874742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:46.874912 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 20:53:46.875037 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 20:53:46.875216 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260619 20:53:46.875909 18834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37117
I20260619 20:53:46.876022 18834 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:46.876230 18834 heartbeater.cc:511] Master 127.0.98.62:37117 requested a full tablet report, sending...
I20260619 20:53:46.876616 18509 ts_manager.cc:194] Registered new tserver with Master: 76990b59d57546d1b3c44e6004bd9479 (127.0.98.2:42037)
I20260619 20:53:46.877022 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37117
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:53:46.899495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37117
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:46.982435 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:46.982664 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:46.982702 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:46.983876 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:46.983947 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:46.983992 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:46.985531 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:46.985780 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:46.986001 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:46.990876 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:46.990948 18839 server_base.cc:1034] running on GCE node
W20260619 20:53:46.990846 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:46.990877 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:46.991204 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:46.991389 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:46.992532 18839 hybrid_clock.cc:648] HybridClock initialized: now 1781902426992522 us; error 26 us; skew 500 ppm
Jun 19 20:53:46 dist-test-slave-0smx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:46.994948 18839 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:53:46.995379 18839 webserver.cc:460] Webserver started at http://127.0.98.3:37931/ using document root <none> and password file <none>
I20260619 20:53:46.995553 18839 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:46.995591 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:46.995688 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:46.996655 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "d180939239864a24878dfbbcd4364a4b"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "bddc15439b8f47c3aeb0d30945e4cd94"
server_key_iv: "e1a726217df33fc52b62dbd8692b3e53"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.996948 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d180939239864a24878dfbbcd4364a4b"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "bddc15439b8f47c3aeb0d30945e4cd94"
server_key_iv: "e1a726217df33fc52b62dbd8692b3e53"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.998394 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:53:46.999351 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:46.999506 18839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:53:46.999593 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "d180939239864a24878dfbbcd4364a4b"
format_stamp: "Formatted at 2026-06-19 20:53:46 on dist-test-slave-0smx"
server_key: "bddc15439b8f47c3aeb0d30945e4cd94"
server_key_iv: "e1a726217df33fc52b62dbd8692b3e53"
server_key_version: "encryptionkey@0"
I20260619 20:53:46.999647 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:47.016561 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:47.018867 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:47.018977 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:47.019161 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:47.019387 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:47.019415 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.019438 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:47.019450 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.025092 18839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34635
I20260619 20:53:47.025185 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34635 every 8 connection(s)
I20260619 20:53:47.025493 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:53:47 dist-test-slave-0smx krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:47.034235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 18839
I20260619 20:53:47.034340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:53:47.034521 392 external_mini_cluster.cc:1428] Setting key 97f63f69b1a56de9849af9236fcee7be
I20260619 20:53:47.035480 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.026248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39681 (local address 127.0.98.62:37117)
0619 20:53:47.026321 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.026324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.026520 (+ 196us) server_negotiation.cc:407] Connection header received
0619 20:53:47.027774 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.027776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.027790 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.027861 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.028497 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.029023 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.030154 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.030374 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.030460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.032686 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.032687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.032687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:47.032696 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.033972 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:47.034452 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:47.034454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:47.034454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:47.034520 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:47.034790 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:47.034791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:47.034791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:47.034917 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.035002 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.035292 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.035384 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:53:47.035655 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.026184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37117 (local address 127.0.98.3:39681)
0619 20:53:47.026387 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.026434 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.027637 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.027933 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.027937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.028192 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:47.028383 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.028388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.029138 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.029139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.030039 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.030042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.030644 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.030644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.030747 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.031173 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:47.031181 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:47.032581 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:47.034075 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:47.034083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:47.034085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:47.034318 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:47.034593 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:47.034594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:47.034594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:47.034714 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:47.034990 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:47.034997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:47.035140 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.035266 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.035421 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260619 20:53:47.036106 18968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37117
I20260619 20:53:47.036216 18968 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:47.036393 18968 heartbeater.cc:511] Master 127.0.98.62:37117 requested a full tablet report, sending...
I20260619 20:53:47.036762 18509 ts_manager.cc:194] Registered new tserver with Master: d180939239864a24878dfbbcd4364a4b (127.0.98.3:34635)
I20260619 20:53:47.037130 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39681
I20260619 20:53:47.046156 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:47.053364 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.047452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.98.62:37117)
0619 20:53:47.047513 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.047515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.047658 (+ 143us) server_negotiation.cc:407] Connection header received
0619 20:53:47.047711 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.047711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.047719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.047767 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.048407 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.048863 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.049679 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.049849 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.049899 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.051048 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.051049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.051050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:47.051055 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.052384 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:47.052725 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:47.052726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:47.052726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:47.052773 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:47.052950 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:47.052950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:47.052951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:47.053067 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.053145 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.053234 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.053278 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:53:47.054993 18509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:47.056507 18509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:47.062469 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.058773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.0.98.3:34635)
0619 20:53:47.058919 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.058924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.059164 (+ 240us) server_negotiation.cc:407] Connection header received
0619 20:53:47.059209 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.059212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.059227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.059310 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.059441 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.060194 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.061329 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.062064 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.062139 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.062193 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.062352 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.062389 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260619 20:53:47.062467 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.058700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34635 (local address 127.0.0.1:48434)
0619 20:53:47.058974 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.059009 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.059090 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.059313 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.059316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.059318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.059375 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.059376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.060287 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.060289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.061213 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.061216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.062156 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.062158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.062239 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.062255 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.062331 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.062383 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":21,"thread_start_us":62,"threads_started":1}
I20260619 20:53:47.062748 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.058699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45015 (local address 127.0.0.1:37838)
0619 20:53:47.058969 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.059006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.059072 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.059429 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.059430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.059433 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.059465 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.059467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.060569 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.060570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.061710 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.061712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.062520 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.062521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.062608 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.062611 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.062660 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.062698 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":68,"threads_started":1}
I20260619 20:53:47.062981 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.058802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.0.98.1:45015)
0619 20:53:47.059126 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.059131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.059141 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:53:47.059185 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.059189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.059205 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.059280 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.059552 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.060465 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.061794 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.062379 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.062437 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.062487 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.062864 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.062912 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":71,"threads_started":1}
I20260619 20:53:47.063373 18902 tablet_service.cc:1467] Processing CreateTablet for tablet af3543c7266d460c9414eaaf96d81420 (DEFAULT_TABLE table=test-table [id=63c63d62f18f41de972da9116733f4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:47.063714 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3543c7266d460c9414eaaf96d81420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.063650 18634 tablet_service.cc:1467] Processing CreateTablet for tablet af3543c7266d460c9414eaaf96d81420 (DEFAULT_TABLE table=test-table [id=63c63d62f18f41de972da9116733f4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:47.063898 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.058896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42037 (local address 127.0.0.1:55140)
0619 20:53:47.059169 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.059178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.059212 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.060293 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.060294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.060296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.060330 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.060331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.061683 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.061684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.062744 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.062746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.063675 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.063676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.063746 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.063751 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.063801 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.063837 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":46,"threads_started":1}
I20260619 20:53:47.063988 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3543c7266d460c9414eaaf96d81420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.064088 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.059117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.98.2:42037)
0619 20:53:47.059298 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.059303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.059311 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:47.059348 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.059350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.059363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.059524 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.060408 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.061558 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.062838 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.063553 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.063625 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.063946 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.063973 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.064017 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260619 20:53:47.066288 18982 tablet_bootstrap.cc:492] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Bootstrap starting.
I20260619 20:53:47.067127 18982 tablet_bootstrap.cc:654] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.067034 18768 tablet_service.cc:1467] Processing CreateTablet for tablet af3543c7266d460c9414eaaf96d81420 (DEFAULT_TABLE table=test-table [id=63c63d62f18f41de972da9116733f4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:47.067350 18768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3543c7266d460c9414eaaf96d81420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.067687 18982 log.cc:824] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.068365 18982 tablet_bootstrap.cc:492] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: No bootstrap required, opened a new log
I20260619 20:53:47.068446 18982 ts_tablet_manager.cc:1397] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:47.069454 18983 tablet_bootstrap.cc:492] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745: Bootstrap starting.
I20260619 20:53:47.070089 18983 tablet_bootstrap.cc:654] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.070042 18982 raft_consensus.cc:348] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.070250 18982 raft_consensus.cc:374] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.070317 18982 raft_consensus.cc:729] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d180939239864a24878dfbbcd4364a4b, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.070377 18985 tablet_bootstrap.cc:492] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Bootstrap starting.
I20260619 20:53:47.070439 18982 consensus_queue.cc:260] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.070700 18983 log.cc:824] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.070747 18982 ts_tablet_manager.cc:1428] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:53:47.070744 18968 heartbeater.cc:503] Master 127.0.98.62:37117 was elected leader, sending a full tablet report...
I20260619 20:53:47.071019 18985 tablet_bootstrap.cc:654] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.071565 18983 tablet_bootstrap.cc:492] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745: No bootstrap required, opened a new log
I20260619 20:53:47.071593 18985 log.cc:824] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.071635 18983 ts_tablet_manager.cc:1397] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:47.072224 18985 tablet_bootstrap.cc:492] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: No bootstrap required, opened a new log
I20260619 20:53:47.072302 18985 ts_tablet_manager.cc:1397] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:47.073154 18983 raft_consensus.cc:348] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.073279 18983 raft_consensus.cc:374] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.073371 18983 raft_consensus.cc:729] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8957652d881744fca12c828872cb6745, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.073504 18983 consensus_queue.cc:260] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.073789 18983 ts_tablet_manager.cc:1428] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:47.073798 18700 heartbeater.cc:503] Master 127.0.98.62:37117 was elected leader, sending a full tablet report...
I20260619 20:53:47.073889 18985 raft_consensus.cc:348] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.074085 18985 raft_consensus.cc:374] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.074137 18985 raft_consensus.cc:729] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76990b59d57546d1b3c44e6004bd9479, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.074251 18985 consensus_queue.cc:260] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.074532 18985 ts_tablet_manager.cc:1428] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:47.074570 18834 heartbeater.cc:503] Master 127.0.98.62:37117 was elected leader, sending a full tablet report...
W20260619 20:53:47.115628 18835 tablet.cc:2369] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:53:47.129495 18989 raft_consensus.cc:482] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:47.129658 18989 raft_consensus.cc:504] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.130031 18989 leader_election.cc:288] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d180939239864a24878dfbbcd4364a4b (127.0.98.3:34635), 76990b59d57546d1b3c44e6004bd9479 (127.0.98.2:42037)
I20260619 20:53:47.133006 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.130330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41787 (local address 127.0.98.2:42037)
0619 20:53:47.130375 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.130377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.130386 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:47.130412 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.130413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.130430 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.130492 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.130650 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.131124 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.132277 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.132650 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.132735 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.132746 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.132902 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.132935 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:53:47.133040 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.130209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42037 (local address 127.0.98.1:41787)
0619 20:53:47.130267 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.130280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.130336 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.130496 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.130496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.130498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.130575 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.130576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.131205 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.131206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.132184 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.132187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.132735 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.132736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.132815 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.132823 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.132884 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.132934 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:53:47.133076 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.130344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41909 (local address 127.0.98.3:34635)
0619 20:53:47.130389 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.130392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.130481 (+ 89us) server_negotiation.cc:407] Connection header received
0619 20:53:47.130560 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.130561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.130570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.130638 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.130789 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.131257 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.132348 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.132705 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.132783 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.132793 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.132964 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.132996 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:53:47.133167 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.130209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34635 (local address 127.0.98.1:41909)
0619 20:53:47.130398 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.130407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.130492 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.130643 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.130643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.130645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.130718 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.130720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.131351 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.131352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.132249 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.132251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.132797 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.132798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.132879 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.132882 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.132943 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.132984 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":13,"thread_start_us":114,"threads_started":1}
I20260619 20:53:47.133463 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3543c7266d460c9414eaaf96d81420" candidate_uuid: "8957652d881744fca12c828872cb6745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76990b59d57546d1b3c44e6004bd9479" is_pre_election: true
I20260619 20:53:47.133497 18922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3543c7266d460c9414eaaf96d81420" candidate_uuid: "8957652d881744fca12c828872cb6745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d180939239864a24878dfbbcd4364a4b" is_pre_election: true
I20260619 20:53:47.133643 18788 raft_consensus.cc:2454] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8957652d881744fca12c828872cb6745 in term 0.
I20260619 20:53:47.133651 18922 raft_consensus.cc:2454] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8957652d881744fca12c828872cb6745 in term 0.
I20260619 20:53:47.133914 18587 leader_election.cc:302] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8957652d881744fca12c828872cb6745, d180939239864a24878dfbbcd4364a4b; no voters:
I20260619 20:53:47.134083 18989 raft_consensus.cc:2781] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:47.134128 18989 raft_consensus.cc:482] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:47.134145 18989 raft_consensus.cc:3037] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:47.134755 18989 raft_consensus.cc:504] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.134891 18989 leader_election.cc:288] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [CANDIDATE]: Term 1 election: Requested vote from peers d180939239864a24878dfbbcd4364a4b (127.0.98.3:34635), 76990b59d57546d1b3c44e6004bd9479 (127.0.98.2:42037)
I20260619 20:53:47.135042 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3543c7266d460c9414eaaf96d81420" candidate_uuid: "8957652d881744fca12c828872cb6745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76990b59d57546d1b3c44e6004bd9479"
I20260619 20:53:47.135041 18922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3543c7266d460c9414eaaf96d81420" candidate_uuid: "8957652d881744fca12c828872cb6745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d180939239864a24878dfbbcd4364a4b"
I20260619 20:53:47.135118 18788 raft_consensus.cc:3037] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:47.135136 18922 raft_consensus.cc:3037] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:47.135792 18922 raft_consensus.cc:2454] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8957652d881744fca12c828872cb6745 in term 1.
I20260619 20:53:47.135799 18788 raft_consensus.cc:2454] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8957652d881744fca12c828872cb6745 in term 1.
I20260619 20:53:47.135980 18587 leader_election.cc:302] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8957652d881744fca12c828872cb6745, d180939239864a24878dfbbcd4364a4b; no voters:
I20260619 20:53:47.136086 18989 raft_consensus.cc:2781] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:47.136250 18989 raft_consensus.cc:686] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [term 1 LEADER]: Becoming Leader. State: Replica: 8957652d881744fca12c828872cb6745, State: Running, Role: LEADER
I20260619 20:53:47.136355 18989 consensus_queue.cc:237] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } }
I20260619 20:53:47.137215 18509 catalog_manager.cc:5495] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8957652d881744fca12c828872cb6745 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8957652d881744fca12c828872cb6745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8957652d881744fca12c828872cb6745" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 } health_report { overall_health: UNKNOWN } } }
I20260619 20:53:47.154320 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.152048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.98.1:45015)
0619 20:53:47.152106 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.152108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.152116 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:47.152195 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.152195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.152212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.152266 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:47.152386 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.152832 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.153624 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.153797 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.153848 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.153975 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:47.154060 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:47.154127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.154177 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.154185 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:53:47.156056 18922 raft_consensus.cc:1261] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 1 FOLLOWER]: Refusing update from remote peer 8957652d881744fca12c828872cb6745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:47.156055 18788 raft_consensus.cc:1261] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 1 FOLLOWER]: Refusing update from remote peer 8957652d881744fca12c828872cb6745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:47.156399 18992 consensus_queue.cc:1035] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d180939239864a24878dfbbcd4364a4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:47.156529 18989 consensus_queue.cc:1035] T af3543c7266d460c9414eaaf96d81420 P 8957652d881744fca12c828872cb6745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76990b59d57546d1b3c44e6004bd9479" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:47.158677 18995 mvcc.cc:205] Tried to move back new op lower bound from 7298672341627904000 to 7298672341550592000. Current Snapshot: MvccSnapshot[applied={T|T < 7298672341627904000}]
I20260619 20:53:47.159214 18996 mvcc.cc:205] Tried to move back new op lower bound from 7298672341627904000 to 7298672341550592000. Current Snapshot: MvccSnapshot[applied={T|T < 7298672341627904000}]
I20260619 20:53:47.159236 18997 mvcc.cc:205] Tried to move back new op lower bound from 7298672341627904000 to 7298672341550592000. Current Snapshot: MvccSnapshot[applied={T|T < 7298672341627904000}]
I20260619 20:53:47.161087 18509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:47.161191 18509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:47.161940 18509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b9ba415a6b0419ba6f57be6d6b0da01: Sending DeleteTablet for 3 replicas of tablet af3543c7266d460c9414eaaf96d81420
I20260619 20:53:47.162245 18902 tablet_service.cc:1514] Processing DeleteTablet for tablet af3543c7266d460c9414eaaf96d81420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48434
I20260619 20:53:47.162256 18768 tablet_service.cc:1514] Processing DeleteTablet for tablet af3543c7266d460c9414eaaf96d81420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55140
I20260619 20:53:47.162258 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet af3543c7266d460c9414eaaf96d81420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37838
I20260619 20:53:47.162454 19006 tablet_replica.cc:330] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: stopping tablet replica
I20260619 20:53:47.162559 19006 raft_consensus.cc:2229] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:47.162659 19006 raft_consensus.cc:2258] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:47.162770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18571
I20260619 20:53:47.163681 19006 ts_tablet_manager.cc:1905] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:47.162804 19005 tablet_replica.cc:330] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: stopping tablet replica
I20260619 20:53:47.164875 19005 raft_consensus.cc:2229] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:47.164996 19005 raft_consensus.cc:2258] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:47.165060 19006 ts_tablet_manager.cc:1918] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:47.165118 19006 log.cc:1196] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/af3543c7266d460c9414eaaf96d81420
I20260619 20:53:47.165390 19006 ts_tablet_manager.cc:1939] T af3543c7266d460c9414eaaf96d81420 P 76990b59d57546d1b3c44e6004bd9479: Deleting consensus metadata
I20260619 20:53:47.165774 19005 ts_tablet_manager.cc:1905] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:47.165803 18496 catalog_manager.cc:4841] TS 76990b59d57546d1b3c44e6004bd9479 (127.0.98.2:42037): tablet af3543c7266d460c9414eaaf96d81420 (table test-table [id=63c63d62f18f41de972da9116733f4d5]) successfully deleted
I20260619 20:53:47.166975 19005 ts_tablet_manager.cc:1918] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:47.167054 19005 log.cc:1196] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/af3543c7266d460c9414eaaf96d81420
I20260619 20:53:47.167321 19005 ts_tablet_manager.cc:1939] T af3543c7266d460c9414eaaf96d81420 P d180939239864a24878dfbbcd4364a4b: Deleting consensus metadata
I20260619 20:53:47.167682 18493 catalog_manager.cc:4841] TS d180939239864a24878dfbbcd4364a4b (127.0.98.3:34635): tablet af3543c7266d460c9414eaaf96d81420 (table test-table [id=63c63d62f18f41de972da9116733f4d5]) successfully deleted
W20260619 20:53:47.168726 18496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45015 (error 108)
I20260619 20:53:47.169133 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.169034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45015 (local address 127.0.0.1:37858)
0619 20:53:47.169081 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.169108 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:53:47.169245 18496 catalog_manager.cc:4584] TS 8957652d881744fca12c828872cb6745 (127.0.98.1:45015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af3543c7266d460c9414eaaf96d81420: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45015: connect: Connection refused (error 111)
I20260619 20:53:47.169401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18705
I20260619 20:53:47.174156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18839
I20260619 20:53:47.178539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 18477
2026-06-19T20:53:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1957 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T20:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:47Z Disabled control of system clock
I20260619 20:53:47.205379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40709
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:47.284895 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:47.285104 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:47.285125 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:47.286311 19012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:47.286353 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:47.286366 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:47.286418 19012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:47.286435 19012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:47.287776 19012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:47.288024 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:47.288311 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:47.293013 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:47.293224 19012 server_base.cc:1034] running on GCE node
W20260619 20:53:47.293154 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:47.293164 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:47.293644 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:47.293924 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:47.295059 19012 hybrid_clock.cc:648] HybridClock initialized: now 1781902427295039 us; error 36 us; skew 500 ppm
I20260619 20:53:47.296308 19012 webserver.cc:460] Webserver started at http://127.0.98.62:40267/ using document root <none> and password file <none>
I20260619 20:53:47.296537 19012 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:47.296583 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:47.296766 19012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:47.297886 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "7df969b426594d19a6aa0b9569c40470"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "802ea743a99fc95a24c1d73a30cb0ac7"
server_key_iv: "6a0b61692b9c5d3d02cbde42fbb87148"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.298234 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "7df969b426594d19a6aa0b9569c40470"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "802ea743a99fc95a24c1d73a30cb0ac7"
server_key_iv: "6a0b61692b9c5d3d02cbde42fbb87148"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.299531 19012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:53:47.300225 19026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.300392 19012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:47.300489 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7df969b426594d19a6aa0b9569c40470"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "802ea743a99fc95a24c1d73a30cb0ac7"
server_key_iv: "6a0b61692b9c5d3d02cbde42fbb87148"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.300565 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:47.323191 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:47.323524 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:47.323688 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:47.327641 19012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40709
I20260619 20:53:47.327672 19078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40709 every 8 connection(s)
I20260619 20:53:47.328011 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:47.328972 19079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.329609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19012
I20260619 20:53:47.329717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:47.329916 392 external_mini_cluster.cc:1428] Setting key aa048d6983b5e3700eebfd101ae120ed
I20260619 20:53:47.331998 19079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470: Bootstrap starting.
I20260619 20:53:47.332661 19079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.332962 19079 log.cc:824] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.333717 19079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470: No bootstrap required, opened a new log
I20260619 20:53:47.335319 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.330456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.0.98.62:40709)
0619 20:53:47.330642 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.330649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.330674 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:53:47.331079 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.331085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.331160 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.331271 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.331526 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.332428 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.333515 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.333948 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.334035 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.334689 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.334692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.334693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:47.334845 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.334876 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:47.334886 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.334933 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.335035 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.335125 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260619 20:53:47.335986 19079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df969b426594d19a6aa0b9569c40470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260619 20:53:47.336140 19079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.336185 19079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df969b426594d19a6aa0b9569c40470, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.336305 19079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df969b426594d19a6aa0b9569c40470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260619 20:53:47.336401 19079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:47.336443 19079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:47.336494 19079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:47.337007 19079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df969b426594d19a6aa0b9569c40470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260619 20:53:47.337126 19079 leader_election.cc:302] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df969b426594d19a6aa0b9569c40470; no voters:
I20260619 20:53:47.337352 19079 leader_election.cc:288] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:47.337392 19084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:47.337539 19084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [term 1 LEADER]: Becoming Leader. State: Replica: 7df969b426594d19a6aa0b9569c40470, State: Running, Role: LEADER
I20260619 20:53:47.337602 19079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:47.337692 19084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df969b426594d19a6aa0b9569c40470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260619 20:53:47.338076 19085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df969b426594d19a6aa0b9569c40470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df969b426594d19a6aa0b9569c40470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } } }
I20260619 20:53:47.338207 19085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:47.338079 19086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df969b426594d19a6aa0b9569c40470. Latest consensus state: current_term: 1 leader_uuid: "7df969b426594d19a6aa0b9569c40470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df969b426594d19a6aa0b9569c40470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } } }
I20260619 20:53:47.338274 19086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:47.338410 19093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:47.338865 19093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:47.340353 19093 catalog_manager.cc:1347] Generated new cluster ID: eb22e9d8252447bd83130827594c5dd6
I20260619 20:53:47.340394 19093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:47.369480 19093 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:47.370034 19093 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:47.374320 19093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470: Generated new TSK 0
I20260619 20:53:47.374501 19093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:47.383502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--builtin_ntp_servers=127.0.98.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:47.463321 19103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:47.463523 19103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:47.463544 19103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:47.464668 19103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:47.464705 19103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:47.464730 19103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:47.466161 19103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:47.466405 19103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:47.466697 19103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:47.471513 19108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:47.471552 19109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:47.471612 19111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:47.471700 19103 server_base.cc:1034] running on GCE node
I20260619 20:53:47.471931 19103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:47.472199 19103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:47.473357 19103 hybrid_clock.cc:648] HybridClock initialized: now 1781902427473343 us; error 30 us; skew 500 ppm
I20260619 20:53:47.474545 19103 webserver.cc:460] Webserver started at http://127.0.98.1:42737/ using document root <none> and password file <none>
I20260619 20:53:47.474803 19103 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:47.474848 19103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:47.475008 19103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:47.476019 19103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "570445139813482d8b393586176f3f8b"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "27f12ebcd07bf66e0a4fc1e5fc67dc0d"
server_key_iv: "93b0b6e8160ffcb616266f55d33bad8b"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.476359 19103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "570445139813482d8b393586176f3f8b"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "27f12ebcd07bf66e0a4fc1e5fc67dc0d"
server_key_iv: "93b0b6e8160ffcb616266f55d33bad8b"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.477634 19103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:53:47.478381 19117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.478549 19103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:53:47.478633 19103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "570445139813482d8b393586176f3f8b"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "27f12ebcd07bf66e0a4fc1e5fc67dc0d"
server_key_iv: "93b0b6e8160ffcb616266f55d33bad8b"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.478708 19103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:47.491127 19103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:47.491376 19103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:47.491509 19103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:47.491707 19103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:47.491997 19103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:47.492049 19103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.492086 19103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:47.492115 19103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.497586 19103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40981
I20260619 20:53:47.497617 19230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40981 every 8 connection(s)
I20260619 20:53:47.497931 19103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:47.498387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19103
I20260619 20:53:47.498494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:53:47.498706 392 external_mini_cluster.cc:1428] Setting key 0ddb0496fa51dc442065ebcfd64df627
I20260619 20:53:47.499714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--builtin_ntp_servers=127.0.98.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:53:47.503948 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.498705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41441 (local address 127.0.98.62:40709)
0619 20:53:47.498746 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.498761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.499025 (+ 264us) server_negotiation.cc:407] Connection header received
0619 20:53:47.499599 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.499600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.499616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.499699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.500410 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.501172 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.502489 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.502675 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.502729 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.503631 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.503632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.503632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:47.503636 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.503650 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:47.503653 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.503703 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.503815 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.503838 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:53:47.504210 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.498644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40709 (local address 127.0.98.1:41441)
0619 20:53:47.498911 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.498954 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.499507 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.499746 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.499749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.499756 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:47.500321 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.500326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.501311 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.501312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.502403 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.502407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.502760 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.502761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.502915 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.503466 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:47.503467 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:47.503509 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:47.503538 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:47.503725 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:47.503732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:47.503735 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.503814 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.503927 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260619 20:53:47.504706 19231 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40709
I20260619 20:53:47.504804 19231 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:47.504966 19231 heartbeater.cc:511] Master 127.0.98.62:40709 requested a full tablet report, sending...
I20260619 20:53:47.505527 19043 ts_manager.cc:194] Registered new tserver with Master: 570445139813482d8b393586176f3f8b (127.0.98.1:40981)
I20260619 20:53:47.506507 19043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41441
W20260619 20:53:47.580056 19234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:47.580250 19234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:47.580271 19234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:47.581423 19234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:47.581460 19234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:47.581485 19234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:47.582788 19234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:47.582976 19234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:47.583182 19234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:47.587970 19242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:47.587980 19239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:47.587980 19240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:47.588074 19234 server_base.cc:1034] running on GCE node
I20260619 20:53:47.588440 19234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:47.588649 19234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:47.589781 19234 hybrid_clock.cc:648] HybridClock initialized: now 1781902427589770 us; error 27 us; skew 500 ppm
I20260619 20:53:47.590968 19234 webserver.cc:460] Webserver started at http://127.0.98.2:36647/ using document root <none> and password file <none>
I20260619 20:53:47.591142 19234 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:47.591180 19234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:47.591281 19234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:47.592218 19234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "2efc0c7788c241d3bbf8f9820a8a4bce"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "9dfbdb04e250c52cf702e22bcc7f4f67"
server_key_iv: "27e62b1dafc8a5e140b36be1189844d5"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.592521 19234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2efc0c7788c241d3bbf8f9820a8a4bce"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "9dfbdb04e250c52cf702e22bcc7f4f67"
server_key_iv: "27e62b1dafc8a5e140b36be1189844d5"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.593828 19234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:47.594515 19248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.594761 19234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:47.594866 19234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "2efc0c7788c241d3bbf8f9820a8a4bce"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "9dfbdb04e250c52cf702e22bcc7f4f67"
server_key_iv: "27e62b1dafc8a5e140b36be1189844d5"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.594952 19234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:47.603024 19234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:47.603291 19234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:47.603428 19234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:47.603634 19234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:47.603945 19234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:47.604000 19234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.604039 19234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:47.604065 19234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.609609 19234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35901
I20260619 20:53:47.609652 19361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35901 every 8 connection(s)
I20260619 20:53:47.609951 19234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:47.614849 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.610567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55969 (local address 127.0.98.62:40709)
0619 20:53:47.610625 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.610627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.610850 (+ 223us) server_negotiation.cc:407] Connection header received
0619 20:53:47.611296 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.611297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.611306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.611356 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.612009 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.612533 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.613598 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.613783 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.613835 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.614566 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.614568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.614568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:47.614571 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.614585 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:47.614588 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.614631 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.614717 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.614738 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:53:47.614971 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.610560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40709 (local address 127.0.98.2:55969)
0619 20:53:47.610721 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.610752 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.611203 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.611418 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.611421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.611429 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:47.611943 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.611946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.612608 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.612609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.613527 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.613529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.613857 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.613858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.613960 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.614434 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:47.614435 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:47.614469 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:47.614496 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:47.614644 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:47.614648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:47.614650 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.614700 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.614792 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260619 20:53:47.614995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19234
I20260619 20:53:47.615088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:53:47.615327 392 external_mini_cluster.cc:1428] Setting key b7d1f12ec87aef06dd28c801e655654d
I20260619 20:53:47.615453 19362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40709
I20260619 20:53:47.615547 19362 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:47.615731 19362 heartbeater.cc:511] Master 127.0.98.62:40709 requested a full tablet report, sending...
I20260619 20:53:47.616115 19043 ts_manager.cc:194] Registered new tserver with Master: 2efc0c7788c241d3bbf8f9820a8a4bce (127.0.98.2:35901)
I20260619 20:53:47.616277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--builtin_ntp_servers=127.0.98.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:53:47.616572 19043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55969
W20260619 20:53:47.695376 19365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:47.695557 19365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:47.695578 19365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:47.696614 19365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:47.696645 19365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:47.696667 19365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:47.697966 19365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:47.698179 19365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:47.698453 19365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:47.703089 19371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:47.703191 19365 server_base.cc:1034] running on GCE node
W20260619 20:53:47.703214 19370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:47.703110 19373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:47.703485 19365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:47.703788 19365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:47.704942 19365 hybrid_clock.cc:648] HybridClock initialized: now 1781902427704927 us; error 27 us; skew 500 ppm
I20260619 20:53:47.706234 19365 webserver.cc:460] Webserver started at http://127.0.98.3:42247/ using document root <none> and password file <none>
I20260619 20:53:47.706487 19365 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:47.706554 19365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:47.706696 19365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:47.707762 19365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "479c9348dd344dd899aec0f9a8f74bd4"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "ec2c775ee8b9565acfd31038e7508305"
server_key_iv: "6f44bdfe708234483adec23e903194f4"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.708125 19365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "479c9348dd344dd899aec0f9a8f74bd4"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "ec2c775ee8b9565acfd31038e7508305"
server_key_iv: "6f44bdfe708234483adec23e903194f4"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.709390 19365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:53:47.710084 19379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.710281 19365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:47.710383 19365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "479c9348dd344dd899aec0f9a8f74bd4"
format_stamp: "Formatted at 2026-06-19 20:53:47 on dist-test-slave-0smx"
server_key: "ec2c775ee8b9565acfd31038e7508305"
server_key_iv: "6f44bdfe708234483adec23e903194f4"
server_key_version: "encryptionkey@0"
I20260619 20:53:47.710460 19365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:47.716869 19365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:47.717133 19365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:47.717271 19365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:47.717504 19365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:47.717794 19365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:47.717849 19365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.717888 19365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:47.717916 19365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:47.723574 19365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44947
I20260619 20:53:47.723619 19492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44947 every 8 connection(s)
I20260619 20:53:47.723951 19365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:53:47.728797 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.724624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60591 (local address 127.0.98.62:40709)
0619 20:53:47.724702 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.724704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.724930 (+ 226us) server_negotiation.cc:407] Connection header received
0619 20:53:47.725423 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.725424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.725435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.725492 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.726117 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.726612 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.727566 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.727762 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.727821 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.728594 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.728594 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.728595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:47.728598 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.728608 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:47.728611 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.728647 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.728719 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.728743 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:53:47.729012 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.724589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40709 (local address 127.0.98.3:60591)
0619 20:53:47.724785 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.724816 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.725269 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.725543 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.725547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.725555 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:47.726040 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.726044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.726716 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.726717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.727481 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.727483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.727844 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.727845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.727963 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.728479 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:47.728480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:47.728507 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:47.728527 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:47.728658 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:47.728660 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:47.728663 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.728707 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.728785 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260619 20:53:47.729557 19493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40709
I20260619 20:53:47.729655 19493 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:47.729825 19493 heartbeater.cc:511] Master 127.0.98.62:40709 requested a full tablet report, sending...
I20260619 20:53:47.730177 19043 ts_manager.cc:194] Registered new tserver with Master: 479c9348dd344dd899aec0f9a8f74bd4 (127.0.98.3:44947)
I20260619 20:53:47.730535 19043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60591
I20260619 20:53:47.730741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19365
I20260619 20:53:47.730861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:53:47.731066 392 external_mini_cluster.cc:1428] Setting key c6065d74c2937c70e5f93a12cd7aa92f
I20260619 20:53:47.731784 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:47.735739 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.733009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.98.62:40709)
0619 20:53:47.733056 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.733057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.733188 (+ 131us) server_negotiation.cc:407] Connection header received
0619 20:53:47.733210 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.733210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.733219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.733267 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:47.733480 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.733926 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.734619 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.734790 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.734837 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.735458 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:47.735459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:47.735459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:47.735463 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:47.735473 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:47.735475 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:47.735542 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.735616 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.735643 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:53:47.737242 19043 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:47.739050 19043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:47.745208 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.741177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44947 (local address 127.0.0.1:47342)
0619 20:53:47.741857 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.741865 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.741933 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.742138 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.742139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.742141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.742181 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.742183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.743094 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.743095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.744101 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.744103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.744911 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.744911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.744982 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.744993 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.745066 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.745125 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":59,"threads_started":1}
I20260619 20:53:47.745219 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.741370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.98.3:44947)
0619 20:53:47.741533 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.741537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.741990 (+ 453us) server_negotiation.cc:407] Connection header received
0619 20:53:47.742037 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.742038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.742050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.742140 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.742234 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.742991 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.744182 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.744800 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.744874 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.744956 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.745087 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.745135 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260619 20:53:47.745219 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.740955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35901 (local address 127.0.0.1:43832)
0619 20:53:47.741116 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.741154 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.741215 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.741491 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.741493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.741496 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.741560 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.741562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.742916 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.742917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.744120 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.744123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.745021 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.745022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.745082 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.745084 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.745129 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.745159 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260619 20:53:47.745428 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.741147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.98.2:35901)
0619 20:53:47.741283 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.741289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.741320 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:53:47.741363 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.741366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.741382 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.741463 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.741661 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.742823 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.744220 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.744812 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.744910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.744995 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.745255 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.745333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260619 20:53:47.746083 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.740981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.0.1:48842)
0619 20:53:47.741342 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:47.741352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:47.741403 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:47.741906 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:47.741907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:47.741909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:47.741945 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.741977 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.743398 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.743399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.744429 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:47.744432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.745835 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.745836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:47.745898 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.745902 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:53:47.745972 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:53:47.746013 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":84,"threads_started":1}
I20260619 20:53:47.746146 19296 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2d357277354b3dbd4a5e4cb4449763 (DEFAULT_TABLE table=test-table [id=79274d538ec141e39671c2e307759be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:47.746109 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:47.741090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.0.98.1:40981)
0619 20:53:47.741578 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:47.741583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:47.741592 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:47.741631 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:47.741634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:47.741649 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:47.741754 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:47.742059 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.743295 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.744517 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:47.745720 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:47.745826 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:47.745917 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:47.745985 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:53:47.746024 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":85,"threads_started":1}
I20260619 20:53:47.746160 19427 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2d357277354b3dbd4a5e4cb4449763 (DEFAULT_TABLE table=test-table [id=79274d538ec141e39671c2e307759be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:47.746516 19427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d357277354b3dbd4a5e4cb4449763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.746521 19296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d357277354b3dbd4a5e4cb4449763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.748961 19507 tablet_bootstrap.cc:492] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Bootstrap starting.
I20260619 20:53:47.749639 19507 tablet_bootstrap.cc:654] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.749619 19165 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2d357277354b3dbd4a5e4cb4449763 (DEFAULT_TABLE table=test-table [id=79274d538ec141e39671c2e307759be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:47.749951 19165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2d357277354b3dbd4a5e4cb4449763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:47.749948 19507 log.cc:824] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.750298 19508 tablet_bootstrap.cc:492] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Bootstrap starting.
I20260619 20:53:47.750936 19508 tablet_bootstrap.cc:654] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.751245 19508 log.cc:824] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.751775 19507 tablet_bootstrap.cc:492] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: No bootstrap required, opened a new log
I20260619 20:53:47.751847 19507 ts_tablet_manager.cc:1397] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 20:53:47.751909 19508 tablet_bootstrap.cc:492] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: No bootstrap required, opened a new log
I20260619 20:53:47.751972 19508 ts_tablet_manager.cc:1397] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:47.753468 19511 tablet_bootstrap.cc:492] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: Bootstrap starting.
I20260619 20:53:47.754163 19511 tablet_bootstrap.cc:654] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:47.754041 19508 raft_consensus.cc:348] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:47.754211 19508 raft_consensus.cc:374] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.754246 19508 raft_consensus.cc:729] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efc0c7788c241d3bbf8f9820a8a4bce, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.754390 19508 consensus_queue.cc:260] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:47.754483 19511 log.cc:824] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:47.754468 19507 raft_consensus.cc:348] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:47.754654 19507 raft_consensus.cc:374] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.754669 19508 ts_tablet_manager.cc:1428] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:53:47.754698 19507 raft_consensus.cc:729] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479c9348dd344dd899aec0f9a8f74bd4, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.754827 19362 heartbeater.cc:503] Master 127.0.98.62:40709 was elected leader, sending a full tablet report...
I20260619 20:53:47.754837 19507 consensus_queue.cc:260] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:47.755128 19493 heartbeater.cc:503] Master 127.0.98.62:40709 was elected leader, sending a full tablet report...
I20260619 20:53:47.755079 19507 ts_tablet_manager.cc:1428] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 20:53:47.755931 19511 tablet_bootstrap.cc:492] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: No bootstrap required, opened a new log
I20260619 20:53:47.755996 19511 ts_tablet_manager.cc:1397] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:53:47.757833 19511 raft_consensus.cc:348] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:47.757973 19511 raft_consensus.cc:374] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:47.758015 19511 raft_consensus.cc:729] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570445139813482d8b393586176f3f8b, State: Initialized, Role: FOLLOWER
I20260619 20:53:47.758121 19511 consensus_queue.cc:260] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:47.758405 19511 ts_tablet_manager.cc:1428] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:53:47.758453 19231 heartbeater.cc:503] Master 127.0.98.62:40709 was elected leader, sending a full tablet report...
W20260619 20:53:47.860483 19363 tablet.cc:2369] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:53:47.974447 19494 tablet.cc:2369] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:53:47.998598 19232 tablet.cc:2369] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:53:48.165248 19512 raft_consensus.cc:482] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:48.165446 19512 raft_consensus.cc:504] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:48.165879 19512 leader_election.cc:288] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 570445139813482d8b393586176f3f8b (127.0.98.1:40981), 479c9348dd344dd899aec0f9a8f74bd4 (127.0.98.3:44947)
I20260619 20:53:48.169905 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:48.166355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60717 (local address 127.0.98.3:44947)
0619 20:53:48.166399 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:48.166413 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:48.166456 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:53:48.166553 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:48.166554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:48.166564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:48.166622 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:48.166839 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.167527 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.169124 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.169558 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.169636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:48.169646 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:48.169805 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:53:48.169843 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:53:48.169929 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:48.166257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44947 (local address 127.0.98.2:60717)
0619 20:53:48.166358 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:48.166370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:48.166467 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:48.166668 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:48.166668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:48.166671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:48.166760 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:48.166762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.167615 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.167616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:48.169001 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:48.169004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.169635 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.169636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:48.169713 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:48.169718 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:48.169796 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:53:48.169836 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":48,"thread_start_us":39,"threads_started":1}
I20260619 20:53:48.170130 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:48.166392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44467 (local address 127.0.98.1:40981)
0619 20:53:48.166445 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:48.166447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:48.166456 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:48.166512 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:48.166514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:48.166526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:48.166590 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:48.166783 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.167528 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.169100 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.169790 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.169874 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:48.169884 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:48.170027 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 20:53:48.170064 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:53:48.170143 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:48.166061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.98.2:44467)
0619 20:53:48.166186 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:48.166199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:48.166253 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:48.166624 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:48.166625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:48.166627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:48.166702 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:48.166705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.167615 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.167616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:48.169001 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:48.169004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.169887 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.169889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:48.169950 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:48.169955 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:48.170021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:53:48.170060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":65,"thread_start_us":83,"threads_started":1}
I20260619 20:53:48.170559 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d357277354b3dbd4a5e4cb4449763" candidate_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479c9348dd344dd899aec0f9a8f74bd4" is_pre_election: true
I20260619 20:53:48.170545 19185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d357277354b3dbd4a5e4cb4449763" candidate_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "570445139813482d8b393586176f3f8b" is_pre_election: true
I20260619 20:53:48.170727 19447 raft_consensus.cc:2454] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efc0c7788c241d3bbf8f9820a8a4bce in term 0.
I20260619 20:53:48.170737 19185 raft_consensus.cc:2454] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efc0c7788c241d3bbf8f9820a8a4bce in term 0.
I20260619 20:53:48.170964 19249 leader_election.cc:302] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efc0c7788c241d3bbf8f9820a8a4bce, 570445139813482d8b393586176f3f8b; no voters:
I20260619 20:53:48.171151 19512 raft_consensus.cc:2781] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:48.171203 19512 raft_consensus.cc:482] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:48.171236 19512 raft_consensus.cc:3037] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:48.171972 19512 raft_consensus.cc:504] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:48.172103 19512 leader_election.cc:288] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [CANDIDATE]: Term 1 election: Requested vote from peers 570445139813482d8b393586176f3f8b (127.0.98.1:40981), 479c9348dd344dd899aec0f9a8f74bd4 (127.0.98.3:44947)
I20260619 20:53:48.172299 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d357277354b3dbd4a5e4cb4449763" candidate_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479c9348dd344dd899aec0f9a8f74bd4"
I20260619 20:53:48.172297 19185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2d357277354b3dbd4a5e4cb4449763" candidate_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "570445139813482d8b393586176f3f8b"
I20260619 20:53:48.172400 19185 raft_consensus.cc:3037] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:48.172397 19447 raft_consensus.cc:3037] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:48.173393 19447 raft_consensus.cc:2454] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efc0c7788c241d3bbf8f9820a8a4bce in term 1.
I20260619 20:53:48.173435 19185 raft_consensus.cc:2454] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efc0c7788c241d3bbf8f9820a8a4bce in term 1.
I20260619 20:53:48.173590 19249 leader_election.cc:302] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efc0c7788c241d3bbf8f9820a8a4bce, 570445139813482d8b393586176f3f8b; no voters:
I20260619 20:53:48.173691 19512 raft_consensus.cc:2781] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:48.173832 19512 raft_consensus.cc:686] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 1 LEADER]: Becoming Leader. State: Replica: 2efc0c7788c241d3bbf8f9820a8a4bce, State: Running, Role: LEADER
I20260619 20:53:48.173924 19512 consensus_queue.cc:237] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } }
I20260619 20:53:48.174779 19043 catalog_manager.cc:5495] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efc0c7788c241d3bbf8f9820a8a4bce (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efc0c7788c241d3bbf8f9820a8a4bce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 } health_report { overall_health: UNKNOWN } } }
I20260619 20:53:48.192340 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:48.189686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.0.98.2:35901)
0619 20:53:48.189759 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:48.189761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:48.189773 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:53:48.189810 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:48.189811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:48.189834 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:48.189930 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:48.190024 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.190860 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.191635 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:48.191951 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:48.192011 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:48.192114 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:48.192196 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:48.192260 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:48.192282 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 20:53:48.192289 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:53:48.194113 19185 raft_consensus.cc:1261] T 3f2d357277354b3dbd4a5e4cb4449763 P 570445139813482d8b393586176f3f8b [term 1 FOLLOWER]: Refusing update from remote peer 2efc0c7788c241d3bbf8f9820a8a4bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:48.194134 19447 raft_consensus.cc:1261] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 1 FOLLOWER]: Refusing update from remote peer 2efc0c7788c241d3bbf8f9820a8a4bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:48.194401 19512 consensus_queue.cc:1035] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "570445139813482d8b393586176f3f8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:48.194517 19518 consensus_queue.cc:1035] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "479c9348dd344dd899aec0f9a8f74bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:48.196913 19522 mvcc.cc:205] Tried to move back new op lower bound from 7298672345880264704 to 7298672345800880128. Current Snapshot: MvccSnapshot[applied={T|T < 7298672345880264704}]
I20260619 20:53:48.197870 19523 mvcc.cc:205] Tried to move back new op lower bound from 7298672345880264704 to 7298672345800880128. Current Snapshot: MvccSnapshot[applied={T|T < 7298672345880264704}]
I20260619 20:53:48.199115 19043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:48.199213 19043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:48.199954 19043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7df969b426594d19a6aa0b9569c40470: Sending DeleteTablet for 3 replicas of tablet 3f2d357277354b3dbd4a5e4cb4449763
I20260619 20:53:48.200310 19296 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2d357277354b3dbd4a5e4cb4449763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:48 UTC) from {username='slave'} at 127.0.0.1:43832
I20260619 20:53:48.200455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19103
I20260619 20:53:48.200521 19427 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2d357277354b3dbd4a5e4cb4449763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:48 UTC) from {username='slave'} at 127.0.0.1:47342
I20260619 20:53:48.201187 19531 tablet_replica.cc:330] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: stopping tablet replica
I20260619 20:53:48.201283 19531 raft_consensus.cc:2229] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:48.201404 19531 raft_consensus.cc:2258] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:48.201805 19531 ts_tablet_manager.cc:1905] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:48.203083 19530 tablet_replica.cc:330] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: stopping tablet replica
I20260619 20:53:48.203203 19530 raft_consensus.cc:2229] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:53:48.203332 19530 raft_consensus.cc:2258] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:48.203739 19531 ts_tablet_manager.cc:1918] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:48.203814 19531 log.cc:1196] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/3f2d357277354b3dbd4a5e4cb4449763
I20260619 20:53:48.203913 19530 ts_tablet_manager.cc:1905] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:48.204080 19531 ts_tablet_manager.cc:1939] T 3f2d357277354b3dbd4a5e4cb4449763 P 479c9348dd344dd899aec0f9a8f74bd4: Deleting consensus metadata
I20260619 20:53:48.204474 19027 catalog_manager.cc:4841] TS 479c9348dd344dd899aec0f9a8f74bd4 (127.0.98.3:44947): tablet 3f2d357277354b3dbd4a5e4cb4449763 (table test-table [id=79274d538ec141e39671c2e307759be7]) successfully deleted
I20260619 20:53:48.205672 19530 ts_tablet_manager.cc:1918] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:48.205796 19530 log.cc:1196] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/3f2d357277354b3dbd4a5e4cb4449763
I20260619 20:53:48.206108 19530 ts_tablet_manager.cc:1939] T 3f2d357277354b3dbd4a5e4cb4449763 P 2efc0c7788c241d3bbf8f9820a8a4bce: Deleting consensus metadata
I20260619 20:53:48.206533 19028 catalog_manager.cc:4841] TS 2efc0c7788c241d3bbf8f9820a8a4bce (127.0.98.2:35901): tablet 3f2d357277354b3dbd4a5e4cb4449763 (table test-table [id=79274d538ec141e39671c2e307759be7]) successfully deleted
W20260619 20:53:48.206657 19027 connection.cc:535] client connection to 127.0.98.1:40981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 20:53:48.206732 19027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:53:48.207178 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:48.207044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.0.1:48844)
0619 20:53:48.207103 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:48.207142 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 20:53:48.207192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19234
W20260619 20:53:48.207384 19027 catalog_manager.cc:4584] TS 570445139813482d8b393586176f3f8b (127.0.98.1:40981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f2d357277354b3dbd4a5e4cb4449763: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40981: connect: Connection refused (error 111)
I20260619 20:53:48.212617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19365
I20260619 20:53:48.217319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19012
2026-06-19T20:53:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1038 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:48 dist-test-slave-0smx krb5kdc[19535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:48 dist-test-slave-0smx krb5kdc[19535](info): set up 2 sockets
Jun 19 20:53:48 dist-test-slave-0smx krb5kdc[19535](info): commencing operation
krb5kdc: starting...
W20260619 20:53:49.241144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:53:49.355572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:49.435196 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:49.435397 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:49.435420 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:49.436546 19548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:49.436585 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:49.436599 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:49.436611 19548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:49.436623 19548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:49.438068 19548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:49.438262 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:49.438434 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:49.443269 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:49.443270 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.443482 19548 server_base.cc:1034] running on GCE node
W20260619 20:53:49.443269 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.443797 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:49.444087 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:49.445253 19548 hybrid_clock.cc:648] HybridClock initialized: now 1781902429445246 us; error 36 us; skew 500 ppm
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:49.450805 19548 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:53:49.451287 19548 webserver.cc:460] Webserver started at http://127.0.98.62:46747/ using document root <none> and password file <none>
I20260619 20:53:49.451510 19548 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:49.451578 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:49.451741 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:49.452797 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "9f4b97179c514c8bb1809a65aa06e6b3"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "bb4c5112ba5edfea6211de6c147fa355"
server_key_iv: "fbc25adfc0089c78da89eebff9e1c16f"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.453174 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f4b97179c514c8bb1809a65aa06e6b3"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "bb4c5112ba5edfea6211de6c147fa355"
server_key_iv: "fbc25adfc0089c78da89eebff9e1c16f"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.454596 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:53:49.455338 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.455511 19548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:53:49.455623 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "9f4b97179c514c8bb1809a65aa06e6b3"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "bb4c5112ba5edfea6211de6c147fa355"
server_key_iv: "fbc25adfc0089c78da89eebff9e1c16f"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.455719 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:49.470193 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:49.472138 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:49.472303 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:49.476665 19548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44523
I20260619 20:53:49.476688 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44523 every 8 connection(s)
I20260619 20:53:49.477041 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:49.477897 19616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:49.480052 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3: Bootstrap starting.
I20260619 20:53:49.480201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19548
I20260619 20:53:49.480347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:49.480537 392 external_mini_cluster.cc:1428] Setting key 91667b389074f5c0483bf4463e55897f
I20260619 20:53:49.480767 19616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:49.481086 19616 log.cc:824] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:49.481765 19616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3: No bootstrap required, opened a new log
I20260619 20:53:49.483537 19616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 20:53:49.483688 19616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:49.483732 19616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4b97179c514c8bb1809a65aa06e6b3, State: Initialized, Role: FOLLOWER
I20260619 20:53:49.483856 19616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 20:53:49.483952 19616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:49.483995 19616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:49.484045 19616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:49.484561 19616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 20:53:49.484676 19616 leader_election.cc:302] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f4b97179c514c8bb1809a65aa06e6b3; no voters:
I20260619 20:53:49.484835 19616 leader_election.cc:288] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:49.484876 19621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:49.484998 19621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 9f4b97179c514c8bb1809a65aa06e6b3, State: Running, Role: LEADER
I20260619 20:53:49.485085 19616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:49.485074 19621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260619 20:53:49.485400 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f4b97179c514c8bb1809a65aa06e6b3. Latest consensus state: current_term: 1 leader_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260619 20:53:49.485479 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:49.485411 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4b97179c514c8bb1809a65aa06e6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260619 20:53:49.485594 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:49.485744 19626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:49.486207 19626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:49.489044 19626 catalog_manager.cc:1347] Generated new cluster ID: b5fd2830a46e428e9cbfc3dca603e33f
I20260619 20:53:49.489099 19626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:49.492153 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.481037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.98.62:44523)
0619 20:53:49.481208 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.481217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.481249 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:53:49.481788 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.481795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.481869 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.482024 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:49.482641 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.483524 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.484462 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.485023 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.485433 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.488868 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:49.488872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:49.488872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:49.488883 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:49.490749 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.491104 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.491107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.491107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.491191 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.491392 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.491393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.491394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.491612 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:49.491770 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.491872 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.492002 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260619 20:53:49.498219 19626 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:49.498868 19626 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:49.508178 19626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3: Generated new TSK 0
I20260619 20:53:49.508348 19626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 20:53:49.537638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:49.624900 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:49.625097 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:49.625120 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:49.626334 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:49.626384 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:49.626410 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:49.627873 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:49.628103 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:49.628346 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:49.633178 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.633257 19642 server_base.cc:1034] running on GCE node
W20260619 20:53:49.633179 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:49.633466 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.633690 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:49.633958 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:49.635111 19642 hybrid_clock.cc:648] HybridClock initialized: now 1781902429635093 us; error 36 us; skew 500 ppm
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:49.637578 19642 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 20:53:49.638032 19642 webserver.cc:460] Webserver started at http://127.0.98.1:45745/ using document root <none> and password file <none>
I20260619 20:53:49.638243 19642 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:49.638288 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:49.638451 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:49.639478 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "77a640d910df4d9a83a105ebc9f71f6c"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "73195a09cb8eec9c78b69735ab80a439"
server_key_iv: "125c946d51bf9f045b0d3a466bbc2fa0"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.639909 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77a640d910df4d9a83a105ebc9f71f6c"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "73195a09cb8eec9c78b69735ab80a439"
server_key_iv: "125c946d51bf9f045b0d3a466bbc2fa0"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.641172 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:49.642009 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.642210 19642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:49.642279 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "77a640d910df4d9a83a105ebc9f71f6c"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "73195a09cb8eec9c78b69735ab80a439"
server_key_iv: "125c946d51bf9f045b0d3a466bbc2fa0"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.642376 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:49.654322 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:49.656553 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:49.656703 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:49.656919 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:49.657214 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:49.657249 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.657270 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:49.657334 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.663301 19642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39515
I20260619 20:53:49.663348 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39515 every 8 connection(s)
I20260619 20:53:49.663656 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:49.672540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19642
I20260619 20:53:49.672676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:53:49.672909 392 external_mini_cluster.cc:1428] Setting key 59337023e1a4c6b6529cbd1f81aa8e13
I20260619 20:53:49.672937 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.664259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59581 (local address 127.0.98.62:44523)
0619 20:53:49.664301 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.664303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.664508 (+ 205us) server_negotiation.cc:407] Connection header received
0619 20:53:49.665499 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.665501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.665514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.665564 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:49.666166 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.666658 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.667647 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.667862 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.667946 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.669980 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:49.669981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:49.669982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:49.669989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:49.671585 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.671944 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.671945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.671946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.671996 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.672201 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.672201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.672202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.672296 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:49.672349 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.672802 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.672874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:53:49.673101 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.664249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.1:59581)
0619 20:53:49.664386 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:49.664419 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:49.665378 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:49.665629 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:49.665633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:49.665889 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:49.666062 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.666066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.666746 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.666747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.667556 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.667558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.667956 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.667957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.668090 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.668528 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:49.668535 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:49.669895 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:49.671655 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:49.671659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:49.671661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:49.671882 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:49.672052 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:49.672053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:49.672053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:49.672151 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:49.672374 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:49.672388 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:49.672545 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 20:53:49.672723 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 20:53:49.672892 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260619 20:53:49.673693 19771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260619 20:53:49.673790 19771 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:49.673959 19771 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260619 20:53:49.674502 19579 ts_manager.cc:194] Registered new tserver with Master: 77a640d910df4d9a83a105ebc9f71f6c (127.0.98.1:39515)
I20260619 20:53:49.675395 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59581
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 20:53:49.696274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:49.775784 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:49.776018 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:49.776055 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:49.777216 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:49.777288 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:49.777364 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:49.778898 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:49.779153 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:49.779383 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:49.784538 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.784584 19776 server_base.cc:1034] running on GCE node
W20260619 20:53:49.784547 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:49.784549 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.784988 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:49.785256 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:49.786417 19776 hybrid_clock.cc:648] HybridClock initialized: now 1781902429786402 us; error 29 us; skew 500 ppm
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:49.788765 19776 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 20:53:49.789247 19776 webserver.cc:460] Webserver started at http://127.0.98.2:37301/ using document root <none> and password file <none>
I20260619 20:53:49.789490 19776 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:49.789561 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:49.789695 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:49.790696 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "e613eed69f18413e99274990aa86a984"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "067e66a6a9d8b1984c06edadf739578d"
server_key_iv: "8cd71607d653192e85474a971dc7b3c2"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.791057 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e613eed69f18413e99274990aa86a984"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "067e66a6a9d8b1984c06edadf739578d"
server_key_iv: "8cd71607d653192e85474a971dc7b3c2"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.792292 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:53:49.792995 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.793169 19776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:49.793273 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "e613eed69f18413e99274990aa86a984"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "067e66a6a9d8b1984c06edadf739578d"
server_key_iv: "8cd71607d653192e85474a971dc7b3c2"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.793355 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:49.807811 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:49.809875 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:49.810039 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:49.810282 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:49.810587 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:49.810640 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.810681 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:49.810706 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.816516 19776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45681
I20260619 20:53:49.816555 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45681 every 8 connection(s)
I20260619 20:53:49.816884 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:49.820590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19776
I20260619 20:53:49.820716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:53:49.820914 392 external_mini_cluster.cc:1428] Setting key 2c544c8c83f29bb2662cc787dd137da7
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:49.826722 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.817614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41217 (local address 127.0.98.62:44523)
0619 20:53:49.817659 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.817661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.817871 (+ 210us) server_negotiation.cc:407] Connection header received
0619 20:53:49.818790 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.818791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.818802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.818848 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:49.819610 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.820103 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.821343 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.821534 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.821621 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.823849 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:49.823851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:49.823851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:49.823859 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:49.825346 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.825787 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.825788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.825789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.825844 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.826108 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.826109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.826109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.826224 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:49.826302 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.826555 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.826633 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:53:49.826937 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.817582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.2:41217)
0619 20:53:49.817743 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:49.817784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:49.818681 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:49.818922 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:49.818926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:49.819311 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:49.819494 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.819499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.820194 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.820195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.821248 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.821251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.821642 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.821643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.821791 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.822325 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:49.822333 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:49.823758 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:49.825449 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:49.825456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:49.825458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:49.825680 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:49.825928 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:49.825929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:49.825929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:49.826037 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:49.826286 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:49.826292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:49.826431 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 20:53:49.826536 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 20:53:49.826686 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260619 20:53:49.827535 19905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260619 20:53:49.827612 19905 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:49.827802 19905 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260619 20:53:49.828173 19579 ts_manager.cc:194] Registered new tserver with Master: e613eed69f18413e99274990aa86a984 (127.0.98.2:45681)
I20260619 20:53:49.828577 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41217
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 20:53:49.846261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:49.925881 19910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:49.926131 19910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:49.926183 19910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:49.927320 19910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:49.927384 19910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:49.927428 19910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:49.928896 19910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:49.929173 19910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:49.929410 19910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:49.934259 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:49.934293 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:49.934293 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:49.934499 19910 server_base.cc:1034] running on GCE node
I20260619 20:53:49.934741 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:49.934991 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:49.936153 19910 hybrid_clock.cc:648] HybridClock initialized: now 1781902429936138 us; error 30 us; skew 500 ppm
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:49.938710 19910 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 20:53:49.939150 19910 webserver.cc:460] Webserver started at http://127.0.98.3:34989/ using document root <none> and password file <none>
I20260619 20:53:49.939352 19910 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:49.939391 19910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:49.939476 19910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:49.940430 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "45fc03098faf4d642b73520c8bd6afb0"
server_key_iv: "14c608c156d280a7f2abafab7a462468"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.940724 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "45fc03098faf4d642b73520c8bd6afb0"
server_key_iv: "14c608c156d280a7f2abafab7a462468"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.942055 19910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:53:49.942727 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.942931 19910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.943009 19910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67"
format_stamp: "Formatted at 2026-06-19 20:53:49 on dist-test-slave-0smx"
server_key: "45fc03098faf4d642b73520c8bd6afb0"
server_key_iv: "14c608c156d280a7f2abafab7a462468"
server_key_version: "encryptionkey@0"
I20260619 20:53:49.943064 19910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:49.949226 19910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:49.951720 19910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:49.951881 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:49.952091 19910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:49.952370 19910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:49.952409 19910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.952430 19910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:49.952471 19910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:49.958259 19910 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36591
I20260619 20:53:49.958297 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36591 every 8 connection(s)
I20260619 20:53:49.958611 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:53:49.960723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 19910
I20260619 20:53:49.960827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:53:49.961032 392 external_mini_cluster.cc:1428] Setting key 6fd62923a585674e01597826a1fc859a
Jun 19 20:53:49 dist-test-slave-0smx krb5kdc[19535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:53:49.968146 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.959368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38089 (local address 127.0.98.62:44523)
0619 20:53:49.959413 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.959416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.959673 (+ 257us) server_negotiation.cc:407] Connection header received
0619 20:53:49.960579 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.960581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.960594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.960676 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:49.961485 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.962193 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.963190 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.963390 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.963461 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.965615 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:49.965616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:49.965617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:49.965624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:49.966849 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.967215 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.967216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.967216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.967268 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.967476 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.967476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.967477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.967587 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:49.967653 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.968010 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.968075 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:53:49.968313 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.959373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.3:38089)
0619 20:53:49.959522 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:49.959569 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:49.960461 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:49.960725 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:49.960728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:49.961116 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:49.961408 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.961415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.962271 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.962271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.963094 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.963096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.963473 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.963474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.963569 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.964106 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:53:49.964114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:49.965532 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:49.966934 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:49.966940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:49.966941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:49.967141 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:49.967325 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:53:49.967326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:53:49.967326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:53:49.967420 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:53:49.967762 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:49.967768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:49.967884 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 20:53:49.967994 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:53:49.968111 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260619 20:53:49.968761 20039 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260619 20:53:49.968849 20039 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:49.968988 20039 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260619 20:53:49.969369 19579 ts_manager.cc:194] Registered new tserver with Master: 3a9cd69f6d4c45b2ae1813dcce7baf67 (127.0.98.3:36591)
I20260619 20:53:49.969751 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38089
I20260619 20:53:49.972565 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:49.979362 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.973913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.98.62:44523)
0619 20:53:49.973955 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.973957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.974111 (+ 154us) server_negotiation.cc:407] Connection header received
0619 20:53:49.974134 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.974135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.974144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.974199 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:49.974731 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.975190 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.975890 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.976078 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.976130 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.977255 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:49.977256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:49.977256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:49.977261 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:49.978455 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.978745 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.978745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.978746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.978790 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:49.978945 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:49.978945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:49.978945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:49.979082 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:49.979154 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.979229 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.979279 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:53:49.980995 19579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:49.982667 19579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:49.989153 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.984638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45681 (local address 127.0.0.1:51488)
0619 20:53:49.985012 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:49.985053 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:49.985131 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:49.985378 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:49.985382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:49.985385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:49.985451 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.985452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.986945 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.986946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.987965 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.987967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.988794 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.988795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.988874 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.988888 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:53:49.988980 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 20:53:49.989037 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":34,"thread_start_us":56,"threads_started":1}
I20260619 20:53:49.989151 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.984828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.98.2:45681)
0619 20:53:49.985060 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.985064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.985133 (+ 69us) server_negotiation.cc:407] Connection header received
0619 20:53:49.985227 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.985229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.985245 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.985372 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:49.985536 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.986844 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.988060 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.988699 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.988775 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.988834 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.989008 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.989051 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":118,"threads_started":1}
I20260619 20:53:49.989499 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.984854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.0.98.3:36591)
0619 20:53:49.985026 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.985030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.985180 (+ 150us) server_negotiation.cc:407] Connection header received
0619 20:53:49.985259 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.985261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.985280 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.985368 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:49.985634 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.986822 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.988397 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.989082 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.989153 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.989204 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.989368 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.989428 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260619 20:53:49.989705 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.984802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.98.1:39515)
0619 20:53:49.985010 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:49.985015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:49.985397 (+ 382us) server_negotiation.cc:407] Connection header received
0619 20:53:49.985452 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:49.985454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:49.985470 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:49.985541 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:49.985904 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.986729 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.988319 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.988923 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.988988 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.989041 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:49.989574 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 20:53:49.989632 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
I20260619 20:53:49.989691 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.984643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39515 (local address 127.0.0.1:38390)
0619 20:53:49.985266 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:49.985275 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:49.985337 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:49.985738 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:49.985739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:49.985741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:49.985804 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.985805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.986868 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.986869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.988224 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.988226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.989395 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.989396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.989470 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.989474 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:53:49.989545 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:53:49.989608 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":82,"thread_start_us":64,"threads_started":1}
I20260619 20:53:49.990039 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:49.984641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36591 (local address 127.0.0.1:53384)
0619 20:53:49.985106 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:49.985114 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:49.985189 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:49.985518 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:49.985519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:49.985521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:49.985573 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.985575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.986947 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.986948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.988315 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:49.988317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:49.989167 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:49.989168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:49.989237 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:49.989243 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:53:49.989624 (+ 381us) client_negotiation.cc:240] Negotiation successful
0619 20:53:49.989672 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":77,"thread_start_us":67,"threads_started":1}
I20260619 20:53:49.990676 19705 tablet_service.cc:1467] Processing CreateTablet for tablet a29dee3efcd7455f8e53362d8434e16b (DEFAULT_TABLE table=test-table [id=d7dbecb9c330430fb6099ab4b344db5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:49.990672 19839 tablet_service.cc:1467] Processing CreateTablet for tablet a29dee3efcd7455f8e53362d8434e16b (DEFAULT_TABLE table=test-table [id=d7dbecb9c330430fb6099ab4b344db5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:49.990751 19973 tablet_service.cc:1467] Processing CreateTablet for tablet a29dee3efcd7455f8e53362d8434e16b (DEFAULT_TABLE table=test-table [id=d7dbecb9c330430fb6099ab4b344db5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:49.991050 19839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29dee3efcd7455f8e53362d8434e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:49.991050 19705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29dee3efcd7455f8e53362d8434e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:49.991070 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29dee3efcd7455f8e53362d8434e16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:49.993966 20053 tablet_bootstrap.cc:492] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Bootstrap starting.
I20260619 20:53:49.994464 20055 tablet_bootstrap.cc:492] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Bootstrap starting.
I20260619 20:53:49.994730 20053 tablet_bootstrap.cc:654] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:49.995062 20055 tablet_bootstrap.cc:654] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:49.995297 20053 log.cc:824] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:49.995664 20055 log.cc:824] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:49.996061 20053 tablet_bootstrap.cc:492] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: No bootstrap required, opened a new log
I20260619 20:53:49.996038 20054 tablet_bootstrap.cc:492] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: Bootstrap starting.
I20260619 20:53:49.996145 20053 ts_tablet_manager.cc:1397] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:49.996676 20054 tablet_bootstrap.cc:654] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:49.997138 20054 log.cc:824] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:49.997663 20055 tablet_bootstrap.cc:492] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: No bootstrap required, opened a new log
I20260619 20:53:49.997715 20055 ts_tablet_manager.cc:1397] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:53:49.997941 20053 raft_consensus.cc:348] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:49.998102 20053 raft_consensus.cc:374] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:49.998150 20053 raft_consensus.cc:729] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9cd69f6d4c45b2ae1813dcce7baf67, State: Initialized, Role: FOLLOWER
I20260619 20:53:49.998306 20053 consensus_queue.cc:260] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:49.998590 20039 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260619 20:53:49.998602 20053 ts_tablet_manager.cc:1428] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:53:49.999256 20055 raft_consensus.cc:348] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:49.999447 20055 raft_consensus.cc:374] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:49.999475 20055 raft_consensus.cc:729] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613eed69f18413e99274990aa86a984, State: Initialized, Role: FOLLOWER
I20260619 20:53:49.999640 20055 consensus_queue.cc:260] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:49.999912 19905 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260619 20:53:49.999920 20055 ts_tablet_manager.cc:1428] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:53:50.000258 20054 tablet_bootstrap.cc:492] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: No bootstrap required, opened a new log
I20260619 20:53:50.000329 20054 ts_tablet_manager.cc:1397] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 20:53:50.002024 20054 raft_consensus.cc:348] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:50.002156 20054 raft_consensus.cc:374] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:50.002177 20054 raft_consensus.cc:729] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a640d910df4d9a83a105ebc9f71f6c, State: Initialized, Role: FOLLOWER
I20260619 20:53:50.002283 20054 consensus_queue.cc:260] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:50.002507 20054 ts_tablet_manager.cc:1428] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 20:53:50.002571 19771 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
W20260619 20:53:50.067535 19906 tablet.cc:2369] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:53:50.078969 20060 raft_consensus.cc:482] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:50.079087 20060 raft_consensus.cc:504] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:50.079442 20060 leader_election.cc:288] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77a640d910df4d9a83a105ebc9f71f6c (127.0.98.1:39515), 3a9cd69f6d4c45b2ae1813dcce7baf67 (127.0.98.3:36591)
I20260619 20:53:50.083372 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.079765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51279 (local address 127.0.98.3:36591)
0619 20:53:50.079823 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.079825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.079907 (+ 82us) server_negotiation.cc:407] Connection header received
0619 20:53:50.079941 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.079942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.079949 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.079990 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.080144 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.080830 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.082405 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.083005 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.083105 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.083116 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.083256 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.083300 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:53:50.083376 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.079686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60325 (local address 127.0.98.1:39515)
0619 20:53:50.079734 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.079736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.079744 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:50.079775 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.079776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.079788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.079855 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.080067 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.080854 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.082405 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.082983 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.083072 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.083085 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.083256 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.083300 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:53:50.083384 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.079615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39515 (local address 127.0.98.2:60325)
0619 20:53:50.079655 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.079667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.079714 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.079881 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.079911 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.079912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.079991 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.079993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.080949 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.080949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.082280 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.082283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.083088 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.083089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.083168 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.083173 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.083235 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.083287 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":22}
I20260619 20:53:50.083406 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.079642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36591 (local address 127.0.98.2:51279)
0619 20:53:50.079811 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.079847 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.079898 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.080009 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.080010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.080011 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.080070 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.080072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.080942 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.080943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.082280 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.082283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.083088 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.083089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.083168 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.083173 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.083235 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.083287 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":89,"thread_start_us":107,"threads_started":1}
I20260619 20:53:50.083799 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29dee3efcd7455f8e53362d8434e16b" candidate_uuid: "e613eed69f18413e99274990aa86a984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" is_pre_election: true
I20260619 20:53:50.083869 19725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29dee3efcd7455f8e53362d8434e16b" candidate_uuid: "e613eed69f18413e99274990aa86a984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a640d910df4d9a83a105ebc9f71f6c" is_pre_election: true
I20260619 20:53:50.083999 19993 raft_consensus.cc:2454] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613eed69f18413e99274990aa86a984 in term 0.
I20260619 20:53:50.084020 19725 raft_consensus.cc:2454] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613eed69f18413e99274990aa86a984 in term 0.
I20260619 20:53:50.084255 19793 leader_election.cc:302] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9cd69f6d4c45b2ae1813dcce7baf67, e613eed69f18413e99274990aa86a984; no voters:
I20260619 20:53:50.084396 20060 raft_consensus.cc:2781] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:50.084455 20060 raft_consensus.cc:482] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:50.084486 20060 raft_consensus.cc:3037] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.085250 20060 raft_consensus.cc:504] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:50.085433 20060 leader_election.cc:288] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [CANDIDATE]: Term 1 election: Requested vote from peers 77a640d910df4d9a83a105ebc9f71f6c (127.0.98.1:39515), 3a9cd69f6d4c45b2ae1813dcce7baf67 (127.0.98.3:36591)
I20260619 20:53:50.085587 19725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29dee3efcd7455f8e53362d8434e16b" candidate_uuid: "e613eed69f18413e99274990aa86a984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a640d910df4d9a83a105ebc9f71f6c"
I20260619 20:53:50.085597 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29dee3efcd7455f8e53362d8434e16b" candidate_uuid: "e613eed69f18413e99274990aa86a984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67"
I20260619 20:53:50.085654 19725 raft_consensus.cc:3037] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.085675 19993 raft_consensus.cc:3037] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.086529 19993 raft_consensus.cc:2454] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613eed69f18413e99274990aa86a984 in term 1.
I20260619 20:53:50.086530 19725 raft_consensus.cc:2454] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613eed69f18413e99274990aa86a984 in term 1.
I20260619 20:53:50.086685 19793 leader_election.cc:302] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9cd69f6d4c45b2ae1813dcce7baf67, e613eed69f18413e99274990aa86a984; no voters:
I20260619 20:53:50.086791 20060 raft_consensus.cc:2781] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:50.086947 20060 raft_consensus.cc:686] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 1 LEADER]: Becoming Leader. State: Replica: e613eed69f18413e99274990aa86a984, State: Running, Role: LEADER
I20260619 20:53:50.087039 20060 consensus_queue.cc:237] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } }
I20260619 20:53:50.087921 19579 catalog_manager.cc:5495] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 reported cstate change: term changed from 0 to 1, leader changed from <none> to e613eed69f18413e99274990aa86a984 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e613eed69f18413e99274990aa86a984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e613eed69f18413e99274990aa86a984" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 } health_report { overall_health: UNKNOWN } } }
I20260619 20:53:50.104931 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.102219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.0.98.2:45681)
0619 20:53:50.102286 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.102288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.102299 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:53:50.102425 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.102426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.102447 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.102540 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:50.102707 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.103205 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.104018 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.104217 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.104313 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.104515 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:50.104634 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:50.104738 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.104765 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.104786 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:53:50.106806 19725 raft_consensus.cc:1261] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 1 FOLLOWER]: Refusing update from remote peer e613eed69f18413e99274990aa86a984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:50.106828 19993 raft_consensus.cc:1261] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 1 FOLLOWER]: Refusing update from remote peer e613eed69f18413e99274990aa86a984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:50.107165 20063 consensus_queue.cc:1035] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a640d910df4d9a83a105ebc9f71f6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:50.107342 20060 consensus_queue.cc:1035] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9cd69f6d4c45b2ae1813dcce7baf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:50.110028 20068 mvcc.cc:205] Tried to move back new op lower bound from 7298672353714163712 to 7298672353636638720. Current Snapshot: MvccSnapshot[applied={T|T < 7298672353714163712}]
I20260619 20:53:50.112047 19580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:50.112155 19580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:50.112902 19580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4b97179c514c8bb1809a65aa06e6b3: Sending DeleteTablet for 3 replicas of tablet a29dee3efcd7455f8e53362d8434e16b
I20260619 20:53:50.113205 19705 tablet_service.cc:1514] Processing DeleteTablet for tablet a29dee3efcd7455f8e53362d8434e16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38390
I20260619 20:53:50.113233 19839 tablet_service.cc:1514] Processing DeleteTablet for tablet a29dee3efcd7455f8e53362d8434e16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51488
I20260619 20:53:50.113544 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet a29dee3efcd7455f8e53362d8434e16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53384
I20260619 20:53:50.113613 20077 tablet_replica.cc:330] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c: stopping tablet replica
I20260619 20:53:50.113706 20077 raft_consensus.cc:2229] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:50.113781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19642
I20260619 20:53:50.113811 20077 raft_consensus.cc:2258] T a29dee3efcd7455f8e53362d8434e16b P 77a640d910df4d9a83a105ebc9f71f6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:50.114311 20076 tablet_replica.cc:330] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: stopping tablet replica
I20260619 20:53:50.114406 20076 raft_consensus.cc:2229] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:53:50.114517 20076 raft_consensus.cc:2258] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:50.114728 20078 tablet_replica.cc:330] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: stopping tablet replica
I20260619 20:53:50.114809 20078 raft_consensus.cc:2229] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:50.114897 20078 raft_consensus.cc:2258] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:50.115335 20078 ts_tablet_manager.cc:1905] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:50.116839 20076 ts_tablet_manager.cc:1905] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:50.117132 20078 ts_tablet_manager.cc:1918] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:50.117204 20078 log.cc:1196] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/a29dee3efcd7455f8e53362d8434e16b
I20260619 20:53:50.117498 20078 ts_tablet_manager.cc:1939] T a29dee3efcd7455f8e53362d8434e16b P 3a9cd69f6d4c45b2ae1813dcce7baf67: Deleting consensus metadata
I20260619 20:53:50.117892 19565 catalog_manager.cc:4841] TS 3a9cd69f6d4c45b2ae1813dcce7baf67 (127.0.98.3:36591): tablet a29dee3efcd7455f8e53362d8434e16b (table test-table [id=d7dbecb9c330430fb6099ab4b344db5f]) successfully deleted
I20260619 20:53:50.118366 20076 ts_tablet_manager.cc:1918] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:50.118422 20076 log.cc:1196] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781902166366061-392-0/minicluster-data/ts-1/wal/wals/a29dee3efcd7455f8e53362d8434e16b
I20260619 20:53:50.118692 20076 ts_tablet_manager.cc:1939] T a29dee3efcd7455f8e53362d8434e16b P e613eed69f18413e99274990aa86a984: Deleting consensus metadata
I20260619 20:53:50.119956 19564 catalog_manager.cc:4841] TS e613eed69f18413e99274990aa86a984 (127.0.98.2:45681): tablet a29dee3efcd7455f8e53362d8434e16b (table test-table [id=d7dbecb9c330430fb6099ab4b344db5f]) successfully deleted
W20260619 20:53:50.121894 19566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39515 (error 108)
I20260619 20:53:50.122184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19776
I20260619 20:53:50.122403 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.122278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39515 (local address 127.0.0.1:38392)
0619 20:53:50.122327 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.122368 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:53:50.122519 19566 catalog_manager.cc:4584] TS 77a640d910df4d9a83a105ebc9f71f6c (127.0.98.1:39515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a29dee3efcd7455f8e53362d8434e16b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39515: connect: Connection refused (error 111)
I20260619 20:53:50.127763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19910
I20260619 20:53:50.131989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 19548
2026-06-19T20:53:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1914 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T20:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:50Z Disabled control of system clock
I20260619 20:53:50.158922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:50.238060 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:50.238271 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:50.238293 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:50.239357 20083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:50.239390 20083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:50.239403 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:50.239416 20083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:50.239428 20083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:50.240692 20083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:50.240885 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:50.241120 20083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:50.245960 20089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:50.245967 20088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:50.245941 20091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.246186 20083 server_base.cc:1034] running on GCE node
I20260619 20:53:50.246425 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:50.246676 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:50.247866 20083 hybrid_clock.cc:648] HybridClock initialized: now 1781902430247844 us; error 37 us; skew 500 ppm
I20260619 20:53:50.249176 20083 webserver.cc:460] Webserver started at http://127.0.98.62:34349/ using document root <none> and password file <none>
I20260619 20:53:50.249470 20083 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:50.249553 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:50.249709 20083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:50.250790 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "d7dcc28c48e4475c89b4a82458210628"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "db87fd52ab114ed50adb325a23724579"
server_key_iv: "54854d2ea005a1da8a3f3cc6240f7917"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.251158 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7dcc28c48e4475c89b4a82458210628"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "db87fd52ab114ed50adb325a23724579"
server_key_iv: "54854d2ea005a1da8a3f3cc6240f7917"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.252458 20083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:50.253180 20097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.253397 20083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:50.253494 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "d7dcc28c48e4475c89b4a82458210628"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "db87fd52ab114ed50adb325a23724579"
server_key_iv: "54854d2ea005a1da8a3f3cc6240f7917"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.253577 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:50.278128 20083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:50.278481 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:50.278645 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:50.282759 20083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34185
I20260619 20:53:50.282812 20149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34185 every 8 connection(s)
I20260619 20:53:50.283160 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:50.284094 20150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:50.286298 20150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628: Bootstrap starting.
I20260619 20:53:50.286883 20150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:50.287199 20150 log.cc:824] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:50.287828 20150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628: No bootstrap required, opened a new log
I20260619 20:53:50.289390 20150 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dcc28c48e4475c89b4a82458210628" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34185 } }
I20260619 20:53:50.289520 20150 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:50.289541 20150 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7dcc28c48e4475c89b4a82458210628, State: Initialized, Role: FOLLOWER
I20260619 20:53:50.289709 20150 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dcc28c48e4475c89b4a82458210628" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34185 } }
I20260619 20:53:50.289770 20150 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:50.289791 20150 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:50.289858 20150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.290364 20150 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dcc28c48e4475c89b4a82458210628" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34185 } }
I20260619 20:53:50.290467 20150 leader_election.cc:302] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7dcc28c48e4475c89b4a82458210628; no voters:
I20260619 20:53:50.290647 20150 leader_election.cc:288] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:50.290686 20153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:50.290865 20153 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [term 1 LEADER]: Becoming Leader. State: Replica: d7dcc28c48e4475c89b4a82458210628, State: Running, Role: LEADER
I20260619 20:53:50.290897 20150 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:50.290961 20153 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dcc28c48e4475c89b4a82458210628" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34185 } }
I20260619 20:53:50.291342 20154 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7dcc28c48e4475c89b4a82458210628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dcc28c48e4475c89b4a82458210628" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34185 } } }
I20260619 20:53:50.291364 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7dcc28c48e4475c89b4a82458210628. Latest consensus state: current_term: 1 leader_uuid: "d7dcc28c48e4475c89b4a82458210628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dcc28c48e4475c89b4a82458210628" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34185 } } }
I20260619 20:53:50.291477 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:50.291694 20154 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:50.291980 20157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:50.292506 20157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:50.293157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20083
I20260619 20:53:50.293260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:50.293519 392 external_mini_cluster.cc:1428] Setting key f1add778813b64ff20f1187009586f53
I20260619 20:53:50.294535 20157 catalog_manager.cc:1347] Generated new cluster ID: 15792316db2c45f68160f25cc0e69e20
I20260619 20:53:50.294591 20157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:50.301837 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.294019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.0.98.62:34185)
0619 20:53:50.297625 (+ 3606us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.297632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.297655 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:53:50.297742 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.297747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.297796 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.297927 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:50.298140 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.299059 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.299872 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.300294 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.300430 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.301129 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:50.301133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:50.301133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:50.301365 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:50.301411 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:50.301422 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:50.301489 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.301570 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.301647 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3528,"thread_start_us":63,"threads_started":1}
I20260619 20:53:50.309615 20157 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:50.310096 20157 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:50.317234 20157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628: Generated new TSK 0
I20260619 20:53:50.317440 20157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:50.319835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34185
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:50.408524 20174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:50.408749 20174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:50.408787 20174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:50.410001 20174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:50.410063 20174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:50.410102 20174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 20:53:50.411469 20174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:50.411725 20174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:50.412021 20174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:50.416837 20180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:50.416870 20179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.417156 20174 server_base.cc:1034] running on GCE node
W20260619 20:53:50.416872 20182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.417481 20174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:50.417758 20174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:50.418908 20174 hybrid_clock.cc:648] HybridClock initialized: now 1781902430418882 us; error 42 us; skew 500 ppm
I20260619 20:53:50.420161 20174 webserver.cc:460] Webserver started at http://127.0.98.1:43455/ using document root <none> and password file <none>
I20260619 20:53:50.420434 20174 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:50.420499 20174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:50.420643 20174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:50.421774 20174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e442ae6b00c4b2a83c613aa50f422ab"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "394a03e45710cccd9b3365c328bf82e9"
server_key_iv: "f5cd0e183b661a0f2ec31126a6c69d84"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.422135 20174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e442ae6b00c4b2a83c613aa50f422ab"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "394a03e45710cccd9b3365c328bf82e9"
server_key_iv: "f5cd0e183b661a0f2ec31126a6c69d84"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.423432 20174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:50.424208 20188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.424422 20174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:50.424532 20174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal
uuid: "2e442ae6b00c4b2a83c613aa50f422ab"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "394a03e45710cccd9b3365c328bf82e9"
server_key_iv: "f5cd0e183b661a0f2ec31126a6c69d84"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.424616 20174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:50.454705 20174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:50.455041 20174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:50.455197 20174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:50.455442 20174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:50.455775 20174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:50.455832 20174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.455878 20174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:50.455907 20174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.461844 20174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33853
I20260619 20:53:50.461884 20301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33853 every 8 connection(s)
I20260619 20:53:50.462216 20174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/data/info.pb
I20260619 20:53:50.467559 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.462970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35019 (local address 127.0.98.62:34185)
0619 20:53:50.463032 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.463034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.463309 (+ 275us) server_negotiation.cc:407] Connection header received
0619 20:53:50.463807 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.463809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.463823 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.463887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:50.464549 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.465058 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.466119 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.466309 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.466370 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.467270 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:50.467271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:50.467272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:50.467276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:50.467289 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:50.467293 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:50.467345 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.467469 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.467494 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:53:50.467797 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.462959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34185 (local address 127.0.98.1:35019)
0619 20:53:50.463185 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.463218 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.463714 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.463958 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.463962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.463970 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:50.464458 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.464462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.465161 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.465162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.466038 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.466041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.466389 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.466390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.466525 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.467126 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:50.467126 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:50.467155 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:50.467177 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:50.467385 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:50.467392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:50.467394 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.467462 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.467540 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260619 20:53:50.468389 20302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34185
I20260619 20:53:50.468497 20302 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:50.468681 20302 heartbeater.cc:511] Master 127.0.98.62:34185 requested a full tablet report, sending...
I20260619 20:53:50.468936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20174
I20260619 20:53:50.469030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-0/wal/instance
I20260619 20:53:50.469213 392 external_mini_cluster.cc:1428] Setting key 136029ce7d3ae6e7b1194fe90295a8c3
I20260619 20:53:50.469228 20114 ts_manager.cc:194] Registered new tserver with Master: 2e442ae6b00c4b2a83c613aa50f422ab (127.0.98.1:33853)
I20260619 20:53:50.469918 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35019
I20260619 20:53:50.470383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34185
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:50.549122 20305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:50.549372 20305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:50.549409 20305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:50.550534 20305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:50.550609 20305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:50.550657 20305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 20:53:50.551993 20305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:50.552266 20305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:50.552560 20305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:50.557451 20311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.557488 20305 server_base.cc:1034] running on GCE node
W20260619 20:53:50.557458 20310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:50.557458 20313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.557852 20305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:50.558094 20305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:50.559226 20305 hybrid_clock.cc:648] HybridClock initialized: now 1781902430559209 us; error 39 us; skew 500 ppm
I20260619 20:53:50.560385 20305 webserver.cc:460] Webserver started at http://127.0.98.2:42121/ using document root <none> and password file <none>
I20260619 20:53:50.560588 20305 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:50.560631 20305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:50.560717 20305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:50.561784 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data/instance:
uuid: "c238d6b892ab48d48ee0636a4ac2e3a7"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "13199c26fcf66231d259d5e75e127624"
server_key_iv: "671b1ab9a4c9a0780176122e7fac3089"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.562068 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c238d6b892ab48d48ee0636a4ac2e3a7"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "13199c26fcf66231d259d5e75e127624"
server_key_iv: "671b1ab9a4c9a0780176122e7fac3089"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.563220 20305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:53:50.563834 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.564007 20305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:50.564088 20305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal
uuid: "c238d6b892ab48d48ee0636a4ac2e3a7"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "13199c26fcf66231d259d5e75e127624"
server_key_iv: "671b1ab9a4c9a0780176122e7fac3089"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.564141 20305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:50.587953 20305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:50.588210 20305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:50.588304 20305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:50.588465 20305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:50.588798 20305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:50.588832 20305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.588886 20305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:50.588912 20305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.594766 20305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40475
I20260619 20:53:50.594882 20432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40475 every 8 connection(s)
I20260619 20:53:50.595180 20305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/data/info.pb
I20260619 20:53:50.600467 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.595932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36939 (local address 127.0.98.62:34185)
0619 20:53:50.596000 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.596002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.596236 (+ 234us) server_negotiation.cc:407] Connection header received
0619 20:53:50.596771 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.596773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.596785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.596838 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:50.597565 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.598052 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.599017 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.599230 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.599313 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.600182 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:50.600183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:50.600184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:50.600189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:50.600203 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:50.600207 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:50.600259 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.600363 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.600390 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:53:50.600699 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.595907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34185 (local address 127.0.98.2:36939)
0619 20:53:50.596118 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.596151 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.596644 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.596926 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.596929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.596936 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:50.597471 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.597475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.598149 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.598150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.598930 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.598932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.599319 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.599320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.599448 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.600023 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:50.600023 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:50.600050 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:50.600071 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:50.600265 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:50.600270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:50.600272 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.600351 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.600450 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260619 20:53:50.601255 20433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34185
I20260619 20:53:50.601372 20433 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:50.601522 20433 heartbeater.cc:511] Master 127.0.98.62:34185 requested a full tablet report, sending...
I20260619 20:53:50.601863 20114 ts_manager.cc:194] Registered new tserver with Master: c238d6b892ab48d48ee0636a4ac2e3a7 (127.0.98.2:40475)
I20260619 20:53:50.602254 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36939
I20260619 20:53:50.605336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20305
I20260619 20:53:50.605451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-1/wal/instance
I20260619 20:53:50.605656 392 external_mini_cluster.cc:1428] Setting key 3933b60cd6dc481bf873ffcd74385c0e
I20260619 20:53:50.606527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34185
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:53:50.685549 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:50.685747 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:50.685770 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:50.686899 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:50.686937 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:50.686961 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 20:53:50.688297 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:50.688520 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:50.688803 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:50.693871 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:50.693876 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.693903 20436 server_base.cc:1034] running on GCE node
W20260619 20:53:50.693877 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:50.694252 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:50.694492 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:50.695652 20436 hybrid_clock.cc:648] HybridClock initialized: now 1781902430695652 us; error 37 us; skew 500 ppm
I20260619 20:53:50.696923 20436 webserver.cc:460] Webserver started at http://127.0.98.3:40287/ using document root <none> and password file <none>
I20260619 20:53:50.697162 20436 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:50.697237 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:50.697384 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:50.698355 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data/instance:
uuid: "102fd38c0c28449aa6ad89361ccf439e"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "7fd1648b2832ce33df042e4ea20b5dfa"
server_key_iv: "51c630b8f42fa1363f92463a3d178fc6"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.698710 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal/instance:
uuid: "102fd38c0c28449aa6ad89361ccf439e"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "7fd1648b2832ce33df042e4ea20b5dfa"
server_key_iv: "51c630b8f42fa1363f92463a3d178fc6"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.699990 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:53:50.700686 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.700886 20436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:50.700980 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal
uuid: "102fd38c0c28449aa6ad89361ccf439e"
format_stamp: "Formatted at 2026-06-19 20:53:50 on dist-test-slave-0smx"
server_key: "7fd1648b2832ce33df042e4ea20b5dfa"
server_key_iv: "51c630b8f42fa1363f92463a3d178fc6"
server_key_version: "encryptionkey@0"
I20260619 20:53:50.701067 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:50.711431 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:50.711702 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:50.711851 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:50.712057 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:53:50.712358 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:53:50.712414 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.712453 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:53:50.712482 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:50.718318 20436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40749
I20260619 20:53:50.718374 20563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40749 every 8 connection(s)
I20260619 20:53:50.718685 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/data/info.pb
I20260619 20:53:50.721227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20436
I20260619 20:53:50.721385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal/instance
I20260619 20:53:50.721602 392 external_mini_cluster.cc:1428] Setting key 55fb4ea10218e419f52e0464882177d0
I20260619 20:53:50.723989 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.719359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44203 (local address 127.0.98.62:34185)
0619 20:53:50.719415 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.719417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.719628 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:53:50.720107 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.720108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.720117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.720175 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:50.720788 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.721480 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.722806 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.722998 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.723052 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.723764 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:50.723765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:50.723765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:50.723769 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:50.723781 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:50.723785 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:50.723827 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.723911 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.723933 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:53:50.724155 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.719310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34185 (local address 127.0.98.3:44203)
0619 20:53:50.719500 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.719540 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.720026 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.720235 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.720238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.720246 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:53:50.720719 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.720724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.721575 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.721576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.722727 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.722730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.723069 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.723070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.723171 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.723641 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:53:50.723641 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:53:50.723668 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:53:50.723688 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:53:50.723839 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:53:50.723843 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:53:50.723845 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.723894 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.723966 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260619 20:53:50.724558 20564 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34185
I20260619 20:53:50.724632 20564 heartbeater.cc:464] Registering TS with master...
I20260619 20:53:50.724812 20564 heartbeater.cc:511] Master 127.0.98.62:34185 requested a full tablet report, sending...
I20260619 20:53:50.725286 20114 ts_manager.cc:194] Registered new tserver with Master: 102fd38c0c28449aa6ad89361ccf439e (127.0.98.3:40749)
I20260619 20:53:50.725678 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44203
I20260619 20:53:50.733144 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:53:50.737097 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.734502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.0.98.62:34185)
0619 20:53:50.734556 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.734557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.734658 (+ 101us) server_negotiation.cc:407] Connection header received
0619 20:53:50.734678 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.734678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.734688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.734735 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:50.734873 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.735335 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.736043 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.736220 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.736269 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.736910 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:50.736910 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:50.736911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:50.736914 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:50.736924 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:50.736927 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:50.736965 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.737023 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.737043 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:53:50.738561 20114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:53:50.740088 20114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:53:50.746280 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.742221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33853 (local address 127.0.0.1:43696)
0619 20:53:50.742580 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.742621 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.742688 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.742997 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.743000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.743003 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.743065 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.743067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.743922 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.743924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.745052 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.745054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.745984 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.745985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.746065 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.746077 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.746145 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.746197 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":78,"threads_started":1}
I20260619 20:53:50.746291 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.742412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.98.1:33853)
0619 20:53:50.742576 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.742580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.742783 (+ 203us) server_negotiation.cc:407] Connection header received
0619 20:53:50.742828 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.742830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.742846 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.742916 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.743143 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.743835 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.745136 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.745893 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.745967 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.746054 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.746155 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.746198 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260619 20:53:50.746408 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.742261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40475 (local address 127.0.0.1:44756)
0619 20:53:50.742588 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.742631 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.742686 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.743017 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.743018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.743020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.743086 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.743087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.744367 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.744368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.745412 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.745414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.746183 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.746184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.746247 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.746250 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.746298 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.746336 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":20,"thread_start_us":91,"threads_started":1}
I20260619 20:53:50.746608 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.742378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.98.2:40475)
0619 20:53:50.742651 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.742654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.742678 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:53:50.742807 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.742810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.742892 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.742958 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.743165 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.744288 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.745487 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.745993 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.746061 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.746152 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.746500 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.746542 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
I20260619 20:53:50.746613 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.742453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40749 (local address 127.0.0.1:52118)
0619 20:53:50.742727 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.742736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.742787 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.743005 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.743006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.743008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.743083 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.743085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.744371 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.744372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.745431 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.745434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.746369 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.746370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.746451 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.746456 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.746511 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.746551 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":75,"threads_started":1}
I20260619 20:53:50.747349 20367 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf9002ae9c54e65aeade8b444e4360f (DEFAULT_TABLE table=test-table [id=d0019736b9cb42f6b395dc77ede1d8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:50.747439 20236 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf9002ae9c54e65aeade8b444e4360f (DEFAULT_TABLE table=test-table [id=d0019736b9cb42f6b395dc77ede1d8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:50.747689 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf9002ae9c54e65aeade8b444e4360f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:50.747794 20236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf9002ae9c54e65aeade8b444e4360f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:50.750387 20578 tablet_bootstrap.cc:492] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: Bootstrap starting.
I20260619 20:53:50.750528 20579 tablet_bootstrap.cc:492] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab: Bootstrap starting.
I20260619 20:53:50.750602 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.742540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.98.3:40749)
0619 20:53:50.742707 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.742711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.742783 (+ 72us) server_negotiation.cc:407] Connection header received
0619 20:53:50.742849 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.742850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.742864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.742941 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.743179 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.744265 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.745520 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.746195 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.746258 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.746333 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.746775 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.746829 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260619 20:53:50.751025 20578 tablet_bootstrap.cc:654] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:50.751171 20579 tablet_bootstrap.cc:654] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:50.751246 20578 log.cc:824] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:50.751266 20498 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf9002ae9c54e65aeade8b444e4360f (DEFAULT_TABLE table=test-table [id=d0019736b9cb42f6b395dc77ede1d8aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:53:50.751467 20579 log.cc:824] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:50.751579 20498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cf9002ae9c54e65aeade8b444e4360f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:50.751909 20578 tablet_bootstrap.cc:492] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: No bootstrap required, opened a new log
I20260619 20:53:50.751953 20578 ts_tablet_manager.cc:1397] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:50.752142 20579 tablet_bootstrap.cc:492] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab: No bootstrap required, opened a new log
I20260619 20:53:50.752207 20579 ts_tablet_manager.cc:1397] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:50.754179 20578 raft_consensus.cc:348] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.754325 20578 raft_consensus.cc:374] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:50.754367 20578 raft_consensus.cc:729] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238d6b892ab48d48ee0636a4ac2e3a7, State: Initialized, Role: FOLLOWER
I20260619 20:53:50.754492 20578 consensus_queue.cc:260] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.754654 20579 raft_consensus.cc:348] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.754786 20433 heartbeater.cc:503] Master 127.0.98.62:34185 was elected leader, sending a full tablet report...
I20260619 20:53:50.754824 20579 raft_consensus.cc:374] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:50.754884 20579 raft_consensus.cc:729] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e442ae6b00c4b2a83c613aa50f422ab, State: Initialized, Role: FOLLOWER
I20260619 20:53:50.755023 20579 consensus_queue.cc:260] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.755060 20583 tablet_bootstrap.cc:492] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Bootstrap starting.
I20260619 20:53:50.755276 20579 ts_tablet_manager.cc:1428] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:53:50.755401 20302 heartbeater.cc:503] Master 127.0.98.62:34185 was elected leader, sending a full tablet report...
I20260619 20:53:50.754781 20578 ts_tablet_manager.cc:1428] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:53:50.755748 20583 tablet_bootstrap.cc:654] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:50.756069 20583 log.cc:824] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:50.756724 20583 tablet_bootstrap.cc:492] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: No bootstrap required, opened a new log
I20260619 20:53:50.756773 20583 ts_tablet_manager.cc:1397] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:53:50.758780 20583 raft_consensus.cc:348] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.758894 20583 raft_consensus.cc:374] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:50.758915 20583 raft_consensus.cc:729] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102fd38c0c28449aa6ad89361ccf439e, State: Initialized, Role: FOLLOWER
I20260619 20:53:50.759045 20583 consensus_queue.cc:260] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.759285 20583 ts_tablet_manager.cc:1428] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:53:50.759351 20564 heartbeater.cc:503] Master 127.0.98.62:34185 was elected leader, sending a full tablet report...
I20260619 20:53:50.785405 20586 raft_consensus.cc:482] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:53:50.785534 20586 raft_consensus.cc:504] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.785943 20586 leader_election.cc:288] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e442ae6b00c4b2a83c613aa50f422ab (127.0.98.1:33853), c238d6b892ab48d48ee0636a4ac2e3a7 (127.0.98.2:40475)
I20260619 20:53:50.789661 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.786244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52135 (local address 127.0.98.2:40475)
0619 20:53:50.786290 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.786292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.786300 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:53:50.786328 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.786329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.786341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.786403 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.786637 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.787243 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.788668 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.789234 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.789347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.789357 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.789522 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.789561 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:53:50.789661 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.786267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60009 (local address 127.0.98.1:33853)
0619 20:53:50.786316 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.786319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.786443 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:53:50.786472 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.786473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.786487 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.786562 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:53:50.786730 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.787300 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.788668 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.789234 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.789348 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.789357 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.789520 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.789559 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:53:50.789688 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.786119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40475 (local address 127.0.98.3:52135)
0619 20:53:50.786180 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.786192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.786236 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.786466 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.786467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.786469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.786552 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.786554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.787315 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.787316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.788564 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.788567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.789347 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.789348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.789443 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.789447 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.789504 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.789555 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":16}
I20260619 20:53:50.789705 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.786162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33853 (local address 127.0.98.3:60009)
0619 20:53:50.786347 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.786355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:53:50.786403 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:53:50.786569 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:53:50.786570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:53:50.786571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:53:50.786657 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.786659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.787364 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.787364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.788564 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:53:50.788567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.789350 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.789350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:53:50.789443 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.789447 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:53:50.789504 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:53:50.789553 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":39,"thread_start_us":116,"threads_started":1}
I20260619 20:53:50.790074 20387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9002ae9c54e65aeade8b444e4360f" candidate_uuid: "102fd38c0c28449aa6ad89361ccf439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" is_pre_election: true
I20260619 20:53:50.790196 20387 raft_consensus.cc:2454] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102fd38c0c28449aa6ad89361ccf439e in term 0.
I20260619 20:53:50.790457 20452 leader_election.cc:302] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102fd38c0c28449aa6ad89361ccf439e, c238d6b892ab48d48ee0636a4ac2e3a7; no voters:
I20260619 20:53:50.790485 20256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9002ae9c54e65aeade8b444e4360f" candidate_uuid: "102fd38c0c28449aa6ad89361ccf439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" is_pre_election: true
I20260619 20:53:50.790618 20256 raft_consensus.cc:2454] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102fd38c0c28449aa6ad89361ccf439e in term 0.
I20260619 20:53:50.790611 20586 raft_consensus.cc:2781] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:53:50.790674 20586 raft_consensus.cc:482] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:53:50.790707 20586 raft_consensus.cc:3037] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.791502 20586 raft_consensus.cc:504] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.791635 20586 leader_election.cc:288] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [CANDIDATE]: Term 1 election: Requested vote from peers 2e442ae6b00c4b2a83c613aa50f422ab (127.0.98.1:33853), c238d6b892ab48d48ee0636a4ac2e3a7 (127.0.98.2:40475)
I20260619 20:53:50.791828 20387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9002ae9c54e65aeade8b444e4360f" candidate_uuid: "102fd38c0c28449aa6ad89361ccf439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7"
I20260619 20:53:50.791887 20256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf9002ae9c54e65aeade8b444e4360f" candidate_uuid: "102fd38c0c28449aa6ad89361ccf439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e442ae6b00c4b2a83c613aa50f422ab"
I20260619 20:53:50.791913 20387 raft_consensus.cc:3037] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.791972 20256 raft_consensus.cc:3037] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:50.792689 20387 raft_consensus.cc:2454] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102fd38c0c28449aa6ad89361ccf439e in term 1.
I20260619 20:53:50.792867 20256 raft_consensus.cc:2454] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102fd38c0c28449aa6ad89361ccf439e in term 1.
I20260619 20:53:50.792908 20452 leader_election.cc:302] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102fd38c0c28449aa6ad89361ccf439e, c238d6b892ab48d48ee0636a4ac2e3a7; no voters:
I20260619 20:53:50.793013 20586 raft_consensus.cc:2781] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:50.793193 20586 raft_consensus.cc:686] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 1 LEADER]: Becoming Leader. State: Replica: 102fd38c0c28449aa6ad89361ccf439e, State: Running, Role: LEADER
I20260619 20:53:50.793329 20586 consensus_queue.cc:237] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } }
I20260619 20:53:50.794097 20114 catalog_manager.cc:5495] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e reported cstate change: term changed from 0 to 1, leader changed from <none> to 102fd38c0c28449aa6ad89361ccf439e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "102fd38c0c28449aa6ad89361ccf439e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "102fd38c0c28449aa6ad89361ccf439e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } health_report { overall_health: UNKNOWN } } }
I20260619 20:53:50.803401 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.801231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.98.3:40749)
0619 20:53:50.801324 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:50.801326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:50.801335 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:53:50.801366 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:50.801367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:50.801383 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:50.801478 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:53:50.801584 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.802017 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.802692 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:50.802890 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:50.802946 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:50.803111 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:53:50.803201 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:53:50.803264 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:50.803299 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:53:50.803314 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:53:50.805091 20387 raft_consensus.cc:1261] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 1 FOLLOWER]: Refusing update from remote peer 102fd38c0c28449aa6ad89361ccf439e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:50.805094 20256 raft_consensus.cc:1261] T 4cf9002ae9c54e65aeade8b444e4360f P 2e442ae6b00c4b2a83c613aa50f422ab [term 1 FOLLOWER]: Refusing update from remote peer 102fd38c0c28449aa6ad89361ccf439e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:53:50.805411 20586 consensus_queue.cc:1035] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238d6b892ab48d48ee0636a4ac2e3a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:50.805567 20588 consensus_queue.cc:1035] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e442ae6b00c4b2a83c613aa50f422ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:53:50.808000 20593 mvcc.cc:205] Tried to move back new op lower bound from 7298672356575207424 to 7298672356530073600. Current Snapshot: MvccSnapshot[applied={T|T < 7298672356575207424}]
I20260619 20:53:50.810134 20113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:50.810225 20113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:53:50.810946 20113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7dcc28c48e4475c89b4a82458210628: Sending DeleteTablet for 3 replicas of tablet 4cf9002ae9c54e65aeade8b444e4360f
I20260619 20:53:50.811280 20367 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf9002ae9c54e65aeade8b444e4360f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:50 UTC) from {username='slave'} at 127.0.0.1:44756
I20260619 20:53:50.811290 20498 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf9002ae9c54e65aeade8b444e4360f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:50 UTC) from {username='slave'} at 127.0.0.1:52118
I20260619 20:53:50.811354 20236 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cf9002ae9c54e65aeade8b444e4360f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:53:50 UTC) from {username='slave'} at 127.0.0.1:43696
I20260619 20:53:50.811542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20174
W20260619 20:53:50.816951 20098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33853 (error 108)
I20260619 20:53:50.817282 20602 tablet_replica.cc:330] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7: stopping tablet replica
I20260619 20:53:50.817435 20602 raft_consensus.cc:2229] T 4cf9002ae9c54e65aeade8b444e4360f P c238d6b892ab48d48ee0636a4ac2e3a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:53:50.817448 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.817327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33853 (local address 127.0.0.1:43704)
0619 20:53:50.817374 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.817414 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:53:50.817464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20305
W20260619 20:53:50.817546 20098 catalog_manager.cc:4584] TS 2e442ae6b00c4b2a83c613aa50f422ab (127.0.98.1:33853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cf9002ae9c54e65aeade8b444e4360f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33853: connect: Connection refused (error 111)
I20260619 20:53:50.818122 20601 tablet_replica.cc:330] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: stopping tablet replica
I20260619 20:53:50.818243 20601 raft_consensus.cc:2229] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:53:50.818394 20601 raft_consensus.cc:2258] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:53:50.819216 20601 ts_tablet_manager.cc:1905] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:53:50.820892 20601 ts_tablet_manager.cc:1918] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:53:50.820967 20601 log.cc:1196] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Deleting WAL directory at /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781902166366061-392-0/minicluster-data/ts-2/wal/wals/4cf9002ae9c54e65aeade8b444e4360f
I20260619 20:53:50.821218 20601 ts_tablet_manager.cc:1939] T 4cf9002ae9c54e65aeade8b444e4360f P 102fd38c0c28449aa6ad89361ccf439e: Deleting consensus metadata
I20260619 20:53:50.821657 20098 catalog_manager.cc:4841] TS 102fd38c0c28449aa6ad89361ccf439e (127.0.98.3:40749): tablet 4cf9002ae9c54e65aeade8b444e4360f (table test-table [id=d0019736b9cb42f6b395dc77ede1d8aa]) successfully deleted
I20260619 20:53:50.822083 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:50.821973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40475 (local address 127.0.0.1:44758)
0619 20:53:50.822021 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:53:50.822047 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:53:50.822163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20436
I20260619 20:53:50.827353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20083
2026-06-19T20:53:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (694 ms)
[----------] 6 tests from EncryptionPolicyTest (8166 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:50 dist-test-slave-0smx krb5kdc[20607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:50 dist-test-slave-0smx krb5kdc[20607](info): set up 2 sockets
Jun 19 20:53:50 dist-test-slave-0smx krb5kdc[20607](info): commencing operation
krb5kdc: starting...
W20260619 20:53:51.857374 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:51 dist-test-slave-0smx krb5kdc[20607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:51.976101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:52.056196 20620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:52.056458 20620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:52.056516 20620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:52.058372 20620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:52.058449 20620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:52.058485 20620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:52.058521 20620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:52.058545 20620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:52.060297 20620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:52.060581 20620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:52.060811 20620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:52.065518 20626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:52.065567 20625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:52.065541 20628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:52.065971 20620 server_base.cc:1034] running on GCE node
I20260619 20:53:52.066259 20620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:52.066560 20620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:52.067699 20620 hybrid_clock.cc:648] HybridClock initialized: now 1781902432067679 us; error 37 us; skew 500 ppm
Jun 19 20:53:52 dist-test-slave-0smx krb5kdc[20607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:52.070107 20620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:52.070542 20620 webserver.cc:460] Webserver started at http://127.0.0.1:35223/ using document root <none> and password file <none>
I20260619 20:53:52.070724 20620 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:52.070762 20620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:52.070848 20620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:52.071822 20620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "d05567bf353a4967b466123b14104491"
format_stamp: "Formatted at 2026-06-19 20:53:52 on dist-test-slave-0smx"
server_key: "e850c3e53f5f1a2502f069b5cacb3c8a"
server_key_iv: "23e8e084ace015ca34ba80caf4686d6f"
server_key_version: "encryptionkey@0"
I20260619 20:53:52.072116 20620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "d05567bf353a4967b466123b14104491"
format_stamp: "Formatted at 2026-06-19 20:53:52 on dist-test-slave-0smx"
server_key: "e850c3e53f5f1a2502f069b5cacb3c8a"
server_key_iv: "23e8e084ace015ca34ba80caf4686d6f"
server_key_version: "encryptionkey@0"
I20260619 20:53:52.073388 20620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:53:52.074082 20635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:52.074251 20620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:52.074355 20620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "d05567bf353a4967b466123b14104491"
format_stamp: "Formatted at 2026-06-19 20:53:52 on dist-test-slave-0smx"
server_key: "e850c3e53f5f1a2502f069b5cacb3c8a"
server_key_iv: "23e8e084ace015ca34ba80caf4686d6f"
server_key_version: "encryptionkey@0"
I20260619 20:53:52.074432 20620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:52.099309 20620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:52.101111 20620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:52.101277 20620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:52.105233 20620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46473
I20260619 20:53:52.105280 20687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46473 every 8 connection(s)
I20260619 20:53:52.105665 20620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:52.106484 20688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:52.108711 20688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491: Bootstrap starting.
I20260619 20:53:52.109339 20688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:52.109642 20688 log.cc:824] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:52.110314 20688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491: No bootstrap required, opened a new log
I20260619 20:53:52.110670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20620
I20260619 20:53:52.110802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:52.110983 392 external_mini_cluster.cc:1428] Setting key c27ae9cf1575300f28da439fe0e116a0
I20260619 20:53:52.112599 20688 raft_consensus.cc:348] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05567bf353a4967b466123b14104491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260619 20:53:52.112728 20688 raft_consensus.cc:374] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:52.112749 20688 raft_consensus.cc:729] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05567bf353a4967b466123b14104491, State: Initialized, Role: FOLLOWER
I20260619 20:53:52.112872 20688 consensus_queue.cc:260] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05567bf353a4967b466123b14104491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260619 20:53:52.112948 20688 raft_consensus.cc:388] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:52.112970 20688 raft_consensus.cc:482] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:52.113020 20688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:52.113788 20688 raft_consensus.cc:504] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05567bf353a4967b466123b14104491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260619 20:53:52.113893 20688 leader_election.cc:302] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d05567bf353a4967b466123b14104491; no voters:
I20260619 20:53:52.114035 20688 leader_election.cc:288] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:52.114099 20693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:52.114248 20693 raft_consensus.cc:686] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [term 1 LEADER]: Becoming Leader. State: Replica: d05567bf353a4967b466123b14104491, State: Running, Role: LEADER
I20260619 20:53:52.114269 20688 sys_catalog.cc:564] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:52.114332 20693 consensus_queue.cc:237] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05567bf353a4967b466123b14104491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } }
I20260619 20:53:52.114632 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d05567bf353a4967b466123b14104491. Latest consensus state: current_term: 1 leader_uuid: "d05567bf353a4967b466123b14104491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05567bf353a4967b466123b14104491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } } }
I20260619 20:53:52.114724 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:52.114643 20694 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d05567bf353a4967b466123b14104491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05567bf353a4967b466123b14104491" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46473 } } }
I20260619 20:53:52.114856 20694 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491 [sys.catalog]: This master's current role is: LEADER
W20260619 20:53:52.116080 20708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:53:52.116139 20708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:53:52.116282 20700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:52.116482 20700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:52.118093 20700 catalog_manager.cc:1347] Generated new cluster ID: 40e0bc4b5d634fcdab0ca1737250d538
I20260619 20:53:52.118155 20700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:53:52 dist-test-slave-0smx krb5kdc[20607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:53:52.122839 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:52.111537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49363 (local address 127.0.0.1:46473)
0619 20:53:52.111720 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:52.111730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:52.111762 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:53:52.112186 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:52.112194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:52.112253 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:52.112400 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:52.113128 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:52.114829 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:52.115672 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:52.116280 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:52.116390 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:52.119119 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:52.119124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:52.119124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:52.119138 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:52.121111 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:52.121552 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:52.121556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:52.121556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:52.121662 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:52.121896 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:52.121897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:52.121897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:52.122209 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:52.122370 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:52.122472 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:53:52.122590 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260619 20:53:52.123936 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:52.132251 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:52.125271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52875 (local address 127.0.0.1:46473)
0619 20:53:52.125323 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:52.125325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:52.125505 (+ 180us) server_negotiation.cc:407] Connection header received
0619 20:53:52.125531 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:52.125532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:52.125542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:52.125603 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:52.126171 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:52.126899 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:52.127845 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:52.128190 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:52.128264 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:52.129577 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:52.129578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:52.129579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:52.129585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:52.131090 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:52.131486 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:52.131487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:52.131488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:52.131548 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:52.131768 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:52.131769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:52.131769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:52.131894 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:52.131996 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:52.132102 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 20:53:52.132162 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:53:52.134768 20700 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:52.135236 20700 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:52.142081 20700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d05567bf353a4967b466123b14104491: Generated new TSK 0
I20260619 20:53:52.142261 20700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:52.144647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20620
W20260619 20:53:52.148528 20605 connection.cc:535] client connection to 127.0.0.1:46473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46473): Connection reset by peer (error 104)
2026-06-19T20:53:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:52 dist-test-slave-0smx krb5kdc[20719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:52 dist-test-slave-0smx krb5kdc[20719](info): set up 2 sockets
Jun 19 20:53:52 dist-test-slave-0smx krb5kdc[20719](info): commencing operation
krb5kdc: starting...
W20260619 20:53:53.173643 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:53 dist-test-slave-0smx krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:53.288511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:53.370882 20732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:53.371097 20732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:53.371121 20732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:53.372228 20732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:53.372269 20732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:53.372287 20732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:53.372300 20732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:53.372314 20732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:53.373811 20732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:53.374014 20732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:53.374181 20732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:53.379354 20740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:53.379302 20737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:53.379323 20732 server_base.cc:1034] running on GCE node
W20260619 20:53:53.379302 20738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:53.379799 20732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:53.380141 20732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:53.381377 20732 hybrid_clock.cc:648] HybridClock initialized: now 1781902433381356 us; error 34 us; skew 500 ppm
Jun 19 20:53:53 dist-test-slave-0smx krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:53.383882 20732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:53.384379 20732 webserver.cc:460] Webserver started at http://127.0.0.1:34703/ using document root <none> and password file <none>
I20260619 20:53:53.384606 20732 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:53.384649 20732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:53.384791 20732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:53.385991 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "7d2574dbb5e74f5c937c4ed91052e06c"
format_stamp: "Formatted at 2026-06-19 20:53:53 on dist-test-slave-0smx"
server_key: "11956f7711a1ab3b3428fbedae766b7a"
server_key_iv: "7c1e07a51cb58392ae5875ed52d321fb"
server_key_version: "encryptionkey@0"
I20260619 20:53:53.386355 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d2574dbb5e74f5c937c4ed91052e06c"
format_stamp: "Formatted at 2026-06-19 20:53:53 on dist-test-slave-0smx"
server_key: "11956f7711a1ab3b3428fbedae766b7a"
server_key_iv: "7c1e07a51cb58392ae5875ed52d321fb"
server_key_version: "encryptionkey@0"
I20260619 20:53:53.387691 20732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:53:53.388438 20747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:53.388638 20732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:53.388708 20732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7d2574dbb5e74f5c937c4ed91052e06c"
format_stamp: "Formatted at 2026-06-19 20:53:53 on dist-test-slave-0smx"
server_key: "11956f7711a1ab3b3428fbedae766b7a"
server_key_iv: "7c1e07a51cb58392ae5875ed52d321fb"
server_key_version: "encryptionkey@0"
I20260619 20:53:53.388796 20732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:53.399487 20732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:53.401283 20732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:53.401508 20732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:53.405560 20732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37257
I20260619 20:53:53.405601 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37257 every 8 connection(s)
I20260619 20:53:53.405943 20732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:53.406721 20800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:53.409008 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c: Bootstrap starting.
I20260619 20:53:53.409641 20800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:53.409927 20800 log.cc:824] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:53.410625 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c: No bootstrap required, opened a new log
I20260619 20:53:53.412163 20800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260619 20:53:53.412320 20800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:53.412365 20800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2574dbb5e74f5c937c4ed91052e06c, State: Initialized, Role: FOLLOWER
I20260619 20:53:53.412539 20800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260619 20:53:53.412635 20800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:53.412679 20800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:53.412729 20800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:53.413161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20732
I20260619 20:53:53.413301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:53.413276 20800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260619 20:53:53.413409 20800 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d2574dbb5e74f5c937c4ed91052e06c; no voters:
I20260619 20:53:53.413582 392 external_mini_cluster.cc:1428] Setting key 3bbf455d3b8b81111e02d1c7845c4150
I20260619 20:53:53.413607 20800 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:53.413637 20803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:53.413792 20803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [term 1 LEADER]: Becoming Leader. State: Replica: 7d2574dbb5e74f5c937c4ed91052e06c, State: Running, Role: LEADER
I20260619 20:53:53.413851 20800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:53.413935 20803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260619 20:53:53.414515 20803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d2574dbb5e74f5c937c4ed91052e06c. Latest consensus state: current_term: 1 leader_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } } }
I20260619 20:53:53.414541 20804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2574dbb5e74f5c937c4ed91052e06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } } }
I20260619 20:53:53.414677 20804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:53.414868 20803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:53.415428 20814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:53.415999 20814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:53.417605 20814 catalog_manager.cc:1347] Generated new cluster ID: 1a3b011f3da54639a9ba0a0083ea4ae6
I20260619 20:53:53.417660 20814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:53.419533 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:53.414181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46323 (local address 127.0.0.1:37257)
0619 20:53:53.414626 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:53.414636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:53.414668 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:53:53.414785 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:53.414792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:53.414848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:53.414987 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:53.415349 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:53.416473 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:53.417617 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:53.418087 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:53.418202 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:53.418923 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:53.418926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:53.418927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:53.419116 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:53.419146 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:53.419157 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:53.419250 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:53.419328 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 20:53:53.419355 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":116,"threads_started":1}
I20260619 20:53:53.420415 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:53.428022 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:53.424947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55749 (local address 127.0.0.1:37257)
0619 20:53:53.424985 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:53.424987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:53.425158 (+ 171us) server_negotiation.cc:407] Connection header received
0619 20:53:53.425218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:53.425219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:53.425259 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:53.425331 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:53.425485 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:53.425993 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:53.426703 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:53.426910 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:53.426984 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:53.427692 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:53.427693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:53.427694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:53.427698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:53.427711 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:53.427715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:53.427769 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:53.427906 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 20:53:53.427919 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:53:53.438882 20814 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:53.439611 20814 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:53.453821 20814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d2574dbb5e74f5c937c4ed91052e06c: Generated new TSK 0
I20260619 20:53:53.454051 20814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:53.457624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20732
W20260619 20:53:53.461653 20717 connection.cc:535] client connection to 127.0.0.1:37257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37257): Connection reset by peer (error 104)
2026-06-19T20:53:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:53 dist-test-slave-0smx krb5kdc[20831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:53 dist-test-slave-0smx krb5kdc[20831](info): set up 2 sockets
Jun 19 20:53:53 dist-test-slave-0smx krb5kdc[20831](info): commencing operation
krb5kdc: starting...
W20260619 20:53:54.493193 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:54 dist-test-slave-0smx krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:54.610445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:54.690577 20844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:54.690784 20844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:54.690807 20844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:54.691885 20844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:54.691926 20844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:54.691939 20844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:54.691952 20844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:54.691964 20844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:54.693462 20844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:54.693646 20844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:54.693821 20844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:54.698719 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:54.698737 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:54.698719 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:54.699097 20844 server_base.cc:1034] running on GCE node
I20260619 20:53:54.699361 20844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:54.699664 20844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:54.700843 20844 hybrid_clock.cc:648] HybridClock initialized: now 1781902434700806 us; error 51 us; skew 500 ppm
Jun 19 20:53:54 dist-test-slave-0smx krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:54.708464 20844 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:54.708968 20844 webserver.cc:460] Webserver started at http://127.0.0.1:38083/ using document root <none> and password file <none>
I20260619 20:53:54.709232 20844 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:54.709327 20844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:54.709493 20844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:54.710597 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "303d6639716d4dfd8762ba4ea7f66316"
format_stamp: "Formatted at 2026-06-19 20:53:54 on dist-test-slave-0smx"
server_key: "0a65b7a77bc30f5846c73fdd943d0a2d"
server_key_iv: "fc2a0c700c45394709dda41757a7c029"
server_key_version: "encryptionkey@0"
I20260619 20:53:54.710968 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "303d6639716d4dfd8762ba4ea7f66316"
format_stamp: "Formatted at 2026-06-19 20:53:54 on dist-test-slave-0smx"
server_key: "0a65b7a77bc30f5846c73fdd943d0a2d"
server_key_iv: "fc2a0c700c45394709dda41757a7c029"
server_key_version: "encryptionkey@0"
I20260619 20:53:54.712311 20844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:53:54.713027 20859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:54.713229 20844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:54.713364 20844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "303d6639716d4dfd8762ba4ea7f66316"
format_stamp: "Formatted at 2026-06-19 20:53:54 on dist-test-slave-0smx"
server_key: "0a65b7a77bc30f5846c73fdd943d0a2d"
server_key_iv: "fc2a0c700c45394709dda41757a7c029"
server_key_version: "encryptionkey@0"
I20260619 20:53:54.713434 20844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:54.716619 20844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:54.718904 20844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:54.719060 20844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:54.722852 20844 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39337
I20260619 20:53:54.722903 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39337 every 8 connection(s)
I20260619 20:53:54.723237 20844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:54.724205 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:54.724799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20844
I20260619 20:53:54.724920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:54.725113 392 external_mini_cluster.cc:1428] Setting key 204f9d8d51e925726ced15f7be172007
I20260619 20:53:54.727484 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316: Bootstrap starting.
I20260619 20:53:54.727686 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:54.725711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37651 (local address 127.0.0.1:39337)
0619 20:53:54.725888 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:54.725897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:54.725928 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:53:54.726443 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:54.726451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:54.726484 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:54.726635 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:54.727233 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:54.727237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:54.727237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:54.727241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:54.727284 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:54.727298 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:54.727372 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:54.727434 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:53:54.727472 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260619 20:53:54.728223 20912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:54.728587 20912 log.cc:824] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:54.728641 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:54.729272 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316: No bootstrap required, opened a new log
I20260619 20:53:54.731451 20912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303d6639716d4dfd8762ba4ea7f66316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260619 20:53:54.731630 20912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:54.731681 20912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303d6639716d4dfd8762ba4ea7f66316, State: Initialized, Role: FOLLOWER
I20260619 20:53:54.731797 20912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303d6639716d4dfd8762ba4ea7f66316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260619 20:53:54.731920 20912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:54.731966 20912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:54.732019 20912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:54.732546 20912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303d6639716d4dfd8762ba4ea7f66316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260619 20:53:54.732668 20912 leader_election.cc:302] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 303d6639716d4dfd8762ba4ea7f66316; no voters:
I20260619 20:53:54.732861 20912 leader_election.cc:288] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:54.732897 20922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:54.733052 20922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [term 1 LEADER]: Becoming Leader. State: Replica: 303d6639716d4dfd8762ba4ea7f66316, State: Running, Role: LEADER
I20260619 20:53:54.733104 20912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:54.733212 20922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303d6639716d4dfd8762ba4ea7f66316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260619 20:53:54.734150 20923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "303d6639716d4dfd8762ba4ea7f66316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303d6639716d4dfd8762ba4ea7f66316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260619 20:53:54.734279 20923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:54.734414 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:54.730022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36067 (local address 127.0.0.1:39337)
0619 20:53:54.730055 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:54.730056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:54.733164 (+ 3108us) server_negotiation.cc:407] Connection header received
0619 20:53:54.733261 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:54.733262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:54.733266 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:54.733341 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:54.734062 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:54.734063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:54.734064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:54.734068 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:54.734081 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:54.734085 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:54.734132 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:54.734265 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 20:53:54.734277 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:53:54.734532 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 303d6639716d4dfd8762ba4ea7f66316. Latest consensus state: current_term: 1 leader_uuid: "303d6639716d4dfd8762ba4ea7f66316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303d6639716d4dfd8762ba4ea7f66316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260619 20:53:54.734757 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316 [sys.catalog]: This master's current role is: LEADER
W20260619 20:53:54.735059 20938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:53:54.735114 20938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:53:54.735183 20936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:54.735366 20936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:54.736735 20936 catalog_manager.cc:1347] Generated new cluster ID: d4f11bb09d374546a041f09aa66c4d92
I20260619 20:53:54.736793 20936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:54.747692 20936 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:54.747881 20936 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:54.751467 20936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 303d6639716d4dfd8762ba4ea7f66316: Generated new TSK 0
I20260619 20:53:54.751648 20936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:54.757026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20844
2026-06-19T20:53:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:54 dist-test-slave-0smx krb5kdc[20943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:54 dist-test-slave-0smx krb5kdc[20943](info): set up 2 sockets
Jun 19 20:53:54 dist-test-slave-0smx krb5kdc[20943](info): commencing operation
krb5kdc: starting...
W20260619 20:53:55.789252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:55 dist-test-slave-0smx krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:55.899817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:55.977136 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:55.977388 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:55.977428 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:55.978567 20956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:55.978644 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:55.978668 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:55.978708 20956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:55.978731 20956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:55.980170 20956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:55.980436 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:55.980654 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:55.985406 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:55.985415 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:55.985412 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:55.985426 20956 server_base.cc:1034] running on GCE node
I20260619 20:53:55.985839 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:55.986122 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:55.987283 20956 hybrid_clock.cc:648] HybridClock initialized: now 1781902435987271 us; error 30 us; skew 500 ppm
Jun 19 20:53:55 dist-test-slave-0smx krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:55.989863 20956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:55.990316 20956 webserver.cc:460] Webserver started at http://127.0.0.1:43159/ using document root <none> and password file <none>
I20260619 20:53:55.990564 20956 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:55.990639 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:55.990792 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:55.991882 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "216abc852208437f8f4127745699c192"
format_stamp: "Formatted at 2026-06-19 20:53:55 on dist-test-slave-0smx"
server_key: "9fd36728dab15bc74ab7e1b5befd8c26"
server_key_iv: "89c1be140f535b59e6d99d6494aba9cd"
server_key_version: "encryptionkey@0"
I20260619 20:53:55.992267 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "216abc852208437f8f4127745699c192"
format_stamp: "Formatted at 2026-06-19 20:53:55 on dist-test-slave-0smx"
server_key: "9fd36728dab15bc74ab7e1b5befd8c26"
server_key_iv: "89c1be140f535b59e6d99d6494aba9cd"
server_key_version: "encryptionkey@0"
I20260619 20:53:55.993657 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:55.994431 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:55.994597 20956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:55.994700 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "216abc852208437f8f4127745699c192"
format_stamp: "Formatted at 2026-06-19 20:53:55 on dist-test-slave-0smx"
server_key: "9fd36728dab15bc74ab7e1b5befd8c26"
server_key_iv: "89c1be140f535b59e6d99d6494aba9cd"
server_key_version: "encryptionkey@0"
I20260619 20:53:55.994787 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:56.013736 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:56.016191 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:56.016362 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:56.020453 20956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42335
I20260619 20:53:56.020515 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42335 every 8 connection(s)
I20260619 20:53:56.020886 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:56.021816 21024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:56.023947 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192: Bootstrap starting.
I20260619 20:53:56.024398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 20956
I20260619 20:53:56.024471 21024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:56.024514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:56.024696 392 external_mini_cluster.cc:1428] Setting key b5f94d02f09b71ed609dcb9f94d7a60c
I20260619 20:53:56.024788 21024 log.cc:824] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:56.025602 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192: No bootstrap required, opened a new log
I20260619 20:53:56.027861 21024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216abc852208437f8f4127745699c192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42335 } }
I20260619 20:53:56.028029 21024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:56.028064 21024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216abc852208437f8f4127745699c192, State: Initialized, Role: FOLLOWER
I20260619 20:53:56.028178 21024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216abc852208437f8f4127745699c192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42335 } }
I20260619 20:53:56.028254 21024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:56.028281 21024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:56.028324 21024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:56.028952 21024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216abc852208437f8f4127745699c192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42335 } }
I20260619 20:53:56.029047 21024 leader_election.cc:302] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 216abc852208437f8f4127745699c192; no voters:
I20260619 20:53:56.029237 21024 leader_election.cc:288] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:56.029260 21029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:56.029412 21029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [term 1 LEADER]: Becoming Leader. State: Replica: 216abc852208437f8f4127745699c192, State: Running, Role: LEADER
I20260619 20:53:56.029562 21029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216abc852208437f8f4127745699c192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42335 } }
I20260619 20:53:56.029568 21024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:56.030746 21031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "216abc852208437f8f4127745699c192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216abc852208437f8f4127745699c192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42335 } } }
I20260619 20:53:56.030833 21031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:56.031036 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:56.031243 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 216abc852208437f8f4127745699c192. Latest consensus state: current_term: 1 leader_uuid: "216abc852208437f8f4127745699c192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216abc852208437f8f4127745699c192" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42335 } } }
I20260619 20:53:56.031334 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:56.031504 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:53:56 dist-test-slave-0smx krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:53:56.036214 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:56.025184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40501 (local address 127.0.0.1:42335)
0619 20:53:56.025381 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:56.025390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:56.025425 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:53:56.025902 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:56.025909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:56.025955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:56.026089 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:56.026855 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:56.027714 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:56.028809 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:56.029402 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:56.031488 (+ 2086us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:56.032557 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:56.032561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:56.032561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:56.032573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:56.034467 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:56.034842 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:56.034844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:56.034845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:56.034963 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:56.035167 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:56.035168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:56.035168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:56.035459 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:56.035615 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:56.035866 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 20:53:56.035995 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260619 20:53:56.036897 21041 catalog_manager.cc:1347] Generated new cluster ID: b47d1eda5bd34b3f9ed4c2d59d8b48d4
I20260619 20:53:56.036950 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:56.041613 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:56.049140 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:56.042946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36113 (local address 127.0.0.1:42335)
0619 20:53:56.042984 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:56.042986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:56.043088 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:53:56.043159 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:56.043160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:56.043170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:56.043226 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:56.043769 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:56.044263 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:56.044963 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:56.045195 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:56.045272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:56.046447 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:56.046448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:56.046449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:56.046455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:56.047837 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:56.048234 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:56.048235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:56.048236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:56.048291 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:56.048504 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:56.048505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:56.048505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:56.048650 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:56.048728 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:56.048949 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 20:53:56.049030 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:53:56.053066 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:56.053845 21041 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:56.057536 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 216abc852208437f8f4127745699c192: Generated new TSK 0
I20260619 20:53:56.057714 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:56.060662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 20956
2026-06-19T20:53:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:56 dist-test-slave-0smx krb5kdc[21055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:56 dist-test-slave-0smx krb5kdc[21055](info): set up 2 sockets
Jun 19 20:53:56 dist-test-slave-0smx krb5kdc[21055](info): commencing operation
krb5kdc: starting...
W20260619 20:53:57.085232 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:57 dist-test-slave-0smx krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:57.197894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:57.277470 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:57.277714 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:57.277750 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:57.278893 21068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:57.278960 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:57.278985 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:57.279006 21068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:57.279034 21068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:57.280481 21068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:57.280753 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:57.280974 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:57.285682 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:57.285676 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:57.285676 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:57.285992 21068 server_base.cc:1034] running on GCE node
I20260619 20:53:57.286252 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:57.286584 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:57.287747 21068 hybrid_clock.cc:648] HybridClock initialized: now 1781902437287733 us; error 26 us; skew 500 ppm
Jun 19 20:53:57 dist-test-slave-0smx krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:57.290141 21068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:57.290570 21068 webserver.cc:460] Webserver started at http://127.0.0.1:34347/ using document root <none> and password file <none>
I20260619 20:53:57.290807 21068 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:57.290874 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:57.291025 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:57.292128 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "02db1730046b41668789007481da5054"
format_stamp: "Formatted at 2026-06-19 20:53:57 on dist-test-slave-0smx"
server_key: "0f16415065224c01e0eaef399b4d7160"
server_key_iv: "c66dc9bbb0b67788060af1f115cc8408"
server_key_version: "encryptionkey@0"
I20260619 20:53:57.292474 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "02db1730046b41668789007481da5054"
format_stamp: "Formatted at 2026-06-19 20:53:57 on dist-test-slave-0smx"
server_key: "0f16415065224c01e0eaef399b4d7160"
server_key_iv: "c66dc9bbb0b67788060af1f115cc8408"
server_key_version: "encryptionkey@0"
I20260619 20:53:57.293812 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:53:57.294473 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:57.294651 21068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:57.294751 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "02db1730046b41668789007481da5054"
format_stamp: "Formatted at 2026-06-19 20:53:57 on dist-test-slave-0smx"
server_key: "0f16415065224c01e0eaef399b4d7160"
server_key_iv: "c66dc9bbb0b67788060af1f115cc8408"
server_key_version: "encryptionkey@0"
I20260619 20:53:57.294819 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:57.313959 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:57.315819 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:57.315989 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:57.319768 21068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44349
I20260619 20:53:57.319800 21135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44349 every 8 connection(s)
I20260619 20:53:57.320158 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:57.320972 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:57.322359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21068
I20260619 20:53:57.322525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:57.322726 392 external_mini_cluster.cc:1428] Setting key 253c6b7a4f08662bcac0c513b1675b4a
I20260619 20:53:57.323567 21136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054: Bootstrap starting.
I20260619 20:53:57.324142 21136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:57.324498 21136 log.cc:824] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:57.325212 21136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054: No bootstrap required, opened a new log
I20260619 20:53:57.327536 21136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02db1730046b41668789007481da5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260619 20:53:57.327687 21136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:57.327723 21136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02db1730046b41668789007481da5054, State: Initialized, Role: FOLLOWER
I20260619 20:53:57.327884 21136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02db1730046b41668789007481da5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260619 20:53:57.327965 21136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:57.328006 21136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:57.328063 21136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:57.328573 21136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02db1730046b41668789007481da5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260619 20:53:57.328697 21136 leader_election.cc:302] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02db1730046b41668789007481da5054; no voters:
I20260619 20:53:57.328887 21136 leader_election.cc:288] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:57.328902 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:57.323348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46029 (local address 127.0.0.1:44349)
0619 20:53:57.323520 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:57.323528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:57.323553 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:53:57.324030 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:57.324037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:57.324084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:57.324231 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:57.324458 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:57.325629 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:57.326734 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:57.327123 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:57.327211 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:57.328099 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:57.328103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:57.328103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:57.328296 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:57.328328 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:57.328340 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:57.328419 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:57.328554 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 20:53:57.328638 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260619 20:53:57.328919 21141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:57.329113 21141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [term 1 LEADER]: Becoming Leader. State: Replica: 02db1730046b41668789007481da5054, State: Running, Role: LEADER
I20260619 20:53:57.329159 21136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:57.329211 21141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02db1730046b41668789007481da5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260619 20:53:57.329629 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02db1730046b41668789007481da5054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02db1730046b41668789007481da5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260619 20:53:57.329761 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:57.329911 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:57.329936 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02db1730046b41668789007481da5054. Latest consensus state: current_term: 1 leader_uuid: "02db1730046b41668789007481da5054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02db1730046b41668789007481da5054" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260619 20:53:57.329988 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:57.330003 21150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:57.330578 21150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:57.332430 21150 catalog_manager.cc:1347] Generated new cluster ID: 2e50a68b994d49c38e5d535ab2e65489
I20260619 20:53:57.332507 21150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:57.335232 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:57.331478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50609 (local address 127.0.0.1:44349)
0619 20:53:57.331573 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:57.331576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:57.331838 (+ 262us) server_negotiation.cc:407] Connection header received
0619 20:53:57.331947 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:57.331948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:57.331960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:57.332031 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:57.332189 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:57.332911 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:57.333935 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:57.334232 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:57.334308 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:57.334981 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:57.334983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:57.334983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:57.334987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:57.335003 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:57.335008 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:57.335069 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:57.335139 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:53:57.335161 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 20:53:57.342199 21150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:57.342921 21150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:57.346478 21150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02db1730046b41668789007481da5054: Generated new TSK 0
I20260619 20:53:57.346652 21150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:57.349824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21068
2026-06-19T20:53:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:57 dist-test-slave-0smx krb5kdc[21167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:57 dist-test-slave-0smx krb5kdc[21167](info): set up 2 sockets
Jun 19 20:53:57 dist-test-slave-0smx krb5kdc[21167](info): commencing operation
krb5kdc: starting...
W20260619 20:53:58.393435 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:58 dist-test-slave-0smx krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:58.508219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:58.588264 21180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:58.588503 21180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:58.588539 21180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:58.589720 21180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:58.589811 21180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:58.589848 21180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:58.589867 21180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:58.589954 21180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:58.591749 21180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:58.591997 21180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:58.592247 21180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:53:58.597124 21180 server_base.cc:1034] running on GCE node
W20260619 20:53:58.597239 21185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:58.597154 21188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:58.597167 21186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:58.597580 21180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:58.597870 21180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:58.599035 21180 hybrid_clock.cc:648] HybridClock initialized: now 1781902438599028 us; error 35 us; skew 500 ppm
Jun 19 20:53:58 dist-test-slave-0smx krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:58.601500 21180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:58.601976 21180 webserver.cc:460] Webserver started at http://127.0.0.1:38309/ using document root <none> and password file <none>
I20260619 20:53:58.602197 21180 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:58.602269 21180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:58.602421 21180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:58.603487 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2"
format_stamp: "Formatted at 2026-06-19 20:53:58 on dist-test-slave-0smx"
server_key: "09aaab80b2c9f8e548ad78ac25c00d4f"
server_key_iv: "6bd635ed6f497c40ebe3cb7c17d090a2"
server_key_version: "encryptionkey@0"
I20260619 20:53:58.603849 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2"
format_stamp: "Formatted at 2026-06-19 20:53:58 on dist-test-slave-0smx"
server_key: "09aaab80b2c9f8e548ad78ac25c00d4f"
server_key_iv: "6bd635ed6f497c40ebe3cb7c17d090a2"
server_key_version: "encryptionkey@0"
I20260619 20:53:58.605254 21180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:53:58.606014 21195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:58.606200 21180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:58.606295 21180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2"
format_stamp: "Formatted at 2026-06-19 20:53:58 on dist-test-slave-0smx"
server_key: "09aaab80b2c9f8e548ad78ac25c00d4f"
server_key_iv: "6bd635ed6f497c40ebe3cb7c17d090a2"
server_key_version: "encryptionkey@0"
I20260619 20:53:58.606395 21180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:58.609665 21180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:58.611609 21180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:58.611781 21180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:58.615746 21180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45141
I20260619 20:53:58.615774 21247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45141 every 8 connection(s)
I20260619 20:53:58.616091 21180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:58.616986 21248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:58.619210 21248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2: Bootstrap starting.
I20260619 20:53:58.619753 21248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:58.620023 21248 log.cc:824] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:58.620682 21248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2: No bootstrap required, opened a new log
I20260619 20:53:58.622228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21180
I20260619 20:53:58.622326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:58.622545 392 external_mini_cluster.cc:1428] Setting key 238081aa98e3d2cf628752860fea2765
I20260619 20:53:58.622484 21248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45141 } }
I20260619 20:53:58.622658 21248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:58.622715 21248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b41dd7b40f84fc5b2e7e34081ade9f2, State: Initialized, Role: FOLLOWER
I20260619 20:53:58.622881 21248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45141 } }
I20260619 20:53:58.622972 21248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:58.623045 21248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:58.623111 21248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:58.624114 21248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45141 } }
I20260619 20:53:58.624231 21248 leader_election.cc:302] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b41dd7b40f84fc5b2e7e34081ade9f2; no voters:
I20260619 20:53:58.624402 21248 leader_election.cc:288] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:58.624464 21253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:58.624605 21253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b41dd7b40f84fc5b2e7e34081ade9f2, State: Running, Role: LEADER
I20260619 20:53:58.624716 21248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:58.624747 21253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45141 } }
I20260619 20:53:58.624902 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:58.622962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41947 (local address 127.0.0.1:45141)
0619 20:53:58.623186 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:58.623195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:58.623225 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:53:58.623328 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:58.623333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:58.623362 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:58.623483 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:58.624165 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:58.624173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:58.624173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:58.624177 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:58.624210 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:58.624220 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:58.624298 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:58.624543 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 20:53:58.624592 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260619 20:53:58.625102 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b41dd7b40f84fc5b2e7e34081ade9f2. Latest consensus state: current_term: 1 leader_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45141 } } }
I20260619 20:53:58.625167 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:58.625268 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b41dd7b40f84fc5b2e7e34081ade9f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45141 } } }
I20260619 20:53:58.625509 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:58.625667 21262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:58.625785 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:58.626953 21262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:58.628391 21262 catalog_manager.cc:1347] Generated new cluster ID: 5828df959ac3428793e7b8f554544696
I20260619 20:53:58.628451 21262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:53:58.635155 21262 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:58.635320 21262 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:58.636267 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:58.635039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54715 (local address 127.0.0.1:45141)
0619 20:53:58.635074 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:58.635076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:58.635273 (+ 197us) server_negotiation.cc:407] Connection header received
0619 20:53:58.635379 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:58.635380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:58.635400 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:58.635464 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:58.636058 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:58.636059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:58.636059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:53:58.636063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:58.636077 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:53:58.636081 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:58.636133 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:58.636199 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 20:53:58.636211 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:53:58.638230 21262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b41dd7b40f84fc5b2e7e34081ade9f2: Generated new TSK 0
I20260619 20:53:58.638404 21262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:58.643761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21180
2026-06-19T20:53:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:58 dist-test-slave-0smx krb5kdc[21279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:58 dist-test-slave-0smx krb5kdc[21279](info): set up 2 sockets
Jun 19 20:53:58 dist-test-slave-0smx krb5kdc[21279](info): commencing operation
krb5kdc: starting...
W20260619 20:53:59.673460 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:53:59 dist-test-slave-0smx krb5kdc[21279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:53:59.787495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:53:59.867599 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:53:59.867810 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:53:59.867833 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:53:59.868916 21292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:53:59.868952 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:53:59.868966 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:53:59.868979 21292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:53:59.868991 21292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:53:59.870414 21292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:53:59.870602 21292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:53:59.870770 21292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:53:59.875460 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:59.875486 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:53:59.875490 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:53:59.875814 21292 server_base.cc:1034] running on GCE node
I20260619 20:53:59.876060 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:53:59.876340 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:53:59.877496 21292 hybrid_clock.cc:648] HybridClock initialized: now 1781902439877477 us; error 37 us; skew 500 ppm
Jun 19 20:53:59 dist-test-slave-0smx krb5kdc[21279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:53:59.883417 21292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:53:59.883905 21292 webserver.cc:460] Webserver started at http://127.0.0.1:34409/ using document root <none> and password file <none>
I20260619 20:53:59.884127 21292 fs_manager.cc:359] Metadata directory not provided
I20260619 20:53:59.884197 21292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:53:59.884346 21292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:53:59.885442 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "9b4905733f0d4b26aa95e31aedbaa853"
format_stamp: "Formatted at 2026-06-19 20:53:59 on dist-test-slave-0smx"
server_key: "37b1dc2aae6bbb6dddc396acd9887699"
server_key_iv: "a53381dd238cb92096ae10d839590889"
server_key_version: "encryptionkey@0"
I20260619 20:53:59.885792 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b4905733f0d4b26aa95e31aedbaa853"
format_stamp: "Formatted at 2026-06-19 20:53:59 on dist-test-slave-0smx"
server_key: "37b1dc2aae6bbb6dddc396acd9887699"
server_key_iv: "a53381dd238cb92096ae10d839590889"
server_key_version: "encryptionkey@0"
I20260619 20:53:59.887053 21292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:53:59.887691 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:53:59.887887 21292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:53:59.887988 21292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "9b4905733f0d4b26aa95e31aedbaa853"
format_stamp: "Formatted at 2026-06-19 20:53:59 on dist-test-slave-0smx"
server_key: "37b1dc2aae6bbb6dddc396acd9887699"
server_key_iv: "a53381dd238cb92096ae10d839590889"
server_key_version: "encryptionkey@0"
I20260619 20:53:59.888058 21292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:53:59.898408 21292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:53:59.900106 21292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:53:59.900259 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:53:59.904335 21292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36495
I20260619 20:53:59.904366 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36495 every 8 connection(s)
I20260619 20:53:59.904702 21292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:53:59.905558 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:53:59.907642 21360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853: Bootstrap starting.
I20260619 20:53:59.908164 21360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853: Neither blocks nor log segments found. Creating new log.
I20260619 20:53:59.908385 21360 log.cc:824] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853: Log is configured to *not* fsync() on all Append() calls
I20260619 20:53:59.908996 21360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853: No bootstrap required, opened a new log
I20260619 20:53:59.910568 21360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4905733f0d4b26aa95e31aedbaa853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36495 } }
I20260619 20:53:59.910702 21360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:53:59.910724 21360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4905733f0d4b26aa95e31aedbaa853, State: Initialized, Role: FOLLOWER
I20260619 20:53:59.910820 21360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4905733f0d4b26aa95e31aedbaa853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36495 } }
I20260619 20:53:59.910868 21360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:53:59.910889 21360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:53:59.910914 21360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:53:59.911391 21360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4905733f0d4b26aa95e31aedbaa853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36495 } }
I20260619 20:53:59.911473 21360 leader_election.cc:302] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4905733f0d4b26aa95e31aedbaa853; no voters:
I20260619 20:53:59.911592 21360 leader_election.cc:288] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:53:59.911660 21363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:53:59.911813 21363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [term 1 LEADER]: Becoming Leader. State: Replica: 9b4905733f0d4b26aa95e31aedbaa853, State: Running, Role: LEADER
I20260619 20:53:59.911823 21360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:53:59.911919 21363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4905733f0d4b26aa95e31aedbaa853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36495 } }
I20260619 20:53:59.912324 21365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4905733f0d4b26aa95e31aedbaa853. Latest consensus state: current_term: 1 leader_uuid: "9b4905733f0d4b26aa95e31aedbaa853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4905733f0d4b26aa95e31aedbaa853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36495 } } }
I20260619 20:53:59.912436 21365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:59.912508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21292
I20260619 20:53:59.912616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:53:59.912815 392 external_mini_cluster.cc:1428] Setting key 1d9bf60084419147f7e9bc86f3a25cb3
I20260619 20:53:59.912317 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4905733f0d4b26aa95e31aedbaa853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4905733f0d4b26aa95e31aedbaa853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36495 } } }
I20260619 20:53:59.912887 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853 [sys.catalog]: This master's current role is: LEADER
I20260619 20:53:59.913022 21367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:53:59.913568 21367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:53:59.915148 21367 catalog_manager.cc:1347] Generated new cluster ID: 3d5e19d313594fc0b4221f843e3208b2
I20260619 20:53:59.915197 21367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:53:59 dist-test-slave-0smx krb5kdc[21279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:53:59.924724 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:59.913260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:32941 (local address 127.0.0.1:36495)
0619 20:53:59.913735 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:59.913745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:59.913779 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:53:59.913891 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:59.913897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:59.913955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:59.914105 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:59.914871 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:59.916270 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:59.917394 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:59.918036 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:59.918120 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:59.921043 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:59.921046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:59.921047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:59.921059 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:59.923056 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:59.923474 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:59.923477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:59.923478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:59.923596 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:59.923826 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:59.923827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:59.923828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:59.924159 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:59.924325 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:59.924410 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:53:59.924513 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":100,"threads_started":1}
I20260619 20:53:59.925928 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:53:59.935985 21367 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:53:59.936609 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:53:59.929539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:41237 (local address 127.0.0.1:36495)
0619 20:53:59.929578 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:53:59.929580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:53:59.929671 (+ 91us) server_negotiation.cc:407] Connection header received
0619 20:53:59.929730 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:53:59.929730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:53:59.929740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:53:59.929788 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:53:59.930387 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:59.931127 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:59.932118 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:53:59.932403 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:53:59.932460 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:53:59.933813 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:53:59.933814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:53:59.933815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:53:59.933821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:53:59.935357 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:59.935780 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:59.935781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:59.935781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:59.935834 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:53:59.936047 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:53:59.936048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:53:59.936049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:53:59.936167 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:53:59.936284 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 20:53:59.936466 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 20:53:59.936526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:53:59.936699 21367 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:53:59.944304 21367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b4905733f0d4b26aa95e31aedbaa853: Generated new TSK 0
I20260619 20:53:59.944480 21367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:53:59.951094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21292
W20260619 20:53:59.955494 21277 connection.cc:535] client connection to 127.0.0.1:36495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36495): Connection reset by peer (error 104)
2026-06-19T20:53:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:53:59 dist-test-slave-0smx krb5kdc[21391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:53:59 dist-test-slave-0smx krb5kdc[21391](info): set up 2 sockets
Jun 19 20:53:59 dist-test-slave-0smx krb5kdc[21391](info): commencing operation
krb5kdc: starting...
W20260619 20:54:00.977522 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:01 dist-test-slave-0smx krb5kdc[21391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:54:01.092201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:01.172233 21404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:01.172482 21404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:01.172525 21404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:01.173715 21404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:01.173779 21404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:01.173804 21404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:01.173827 21404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:01.173846 21404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:01.175304 21404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:01.175575 21404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:01.175863 21404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:01.180634 21410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:01.180691 21412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:01.180727 21409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:01.180984 21404 server_base.cc:1034] running on GCE node
I20260619 20:54:01.181237 21404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:01.181582 21404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:01.182746 21404 hybrid_clock.cc:648] HybridClock initialized: now 1781902441182733 us; error 44 us; skew 500 ppm
Jun 19 20:54:01 dist-test-slave-0smx krb5kdc[21391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:01.185155 21404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:54:01.185622 21404 webserver.cc:460] Webserver started at http://127.0.0.1:46079/ using document root <none> and password file <none>
I20260619 20:54:01.185842 21404 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:01.185909 21404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:01.186070 21404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:01.187130 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "7a39333fc64049a59f910f1fbc86a30c"
format_stamp: "Formatted at 2026-06-19 20:54:01 on dist-test-slave-0smx"
server_key: "738526a4524776059097cfdd5be6d781"
server_key_iv: "34a20907e9ff0a07fc676e21f6873ae2"
server_key_version: "encryptionkey@0"
I20260619 20:54:01.187420 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a39333fc64049a59f910f1fbc86a30c"
format_stamp: "Formatted at 2026-06-19 20:54:01 on dist-test-slave-0smx"
server_key: "738526a4524776059097cfdd5be6d781"
server_key_iv: "34a20907e9ff0a07fc676e21f6873ae2"
server_key_version: "encryptionkey@0"
I20260619 20:54:01.188637 21404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:01.189399 21419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:01.189594 21404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:54:01.189664 21404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7a39333fc64049a59f910f1fbc86a30c"
format_stamp: "Formatted at 2026-06-19 20:54:01 on dist-test-slave-0smx"
server_key: "738526a4524776059097cfdd5be6d781"
server_key_iv: "34a20907e9ff0a07fc676e21f6873ae2"
server_key_version: "encryptionkey@0"
I20260619 20:54:01.189749 21404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:01.211473 21404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:01.213547 21404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:01.213703 21404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:01.217640 21404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34729
I20260619 20:54:01.217696 21471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34729 every 8 connection(s)
I20260619 20:54:01.218050 21404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:01.218873 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:01.221092 21472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c: Bootstrap starting.
I20260619 20:54:01.221689 21472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:01.221975 21472 log.cc:824] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:01.222645 21472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c: No bootstrap required, opened a new log
I20260619 20:54:01.224220 21472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39333fc64049a59f910f1fbc86a30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34729 } }
I20260619 20:54:01.224367 21472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:01.224390 21472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a39333fc64049a59f910f1fbc86a30c, State: Initialized, Role: FOLLOWER
I20260619 20:54:01.224581 21472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39333fc64049a59f910f1fbc86a30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34729 } }
I20260619 20:54:01.224663 21472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:01.224712 21472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:01.224761 21472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:01.225255 21472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39333fc64049a59f910f1fbc86a30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34729 } }
I20260619 20:54:01.225394 21472 leader_election.cc:302] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a39333fc64049a59f910f1fbc86a30c; no voters:
I20260619 20:54:01.225576 21472 leader_election.cc:288] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:01.225622 21475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:01.225786 21475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [term 1 LEADER]: Becoming Leader. State: Replica: 7a39333fc64049a59f910f1fbc86a30c, State: Running, Role: LEADER
I20260619 20:54:01.225831 21472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:01.225917 21475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39333fc64049a59f910f1fbc86a30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34729 } }
I20260619 20:54:01.226302 21476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a39333fc64049a59f910f1fbc86a30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39333fc64049a59f910f1fbc86a30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34729 } } }
I20260619 20:54:01.226308 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a39333fc64049a59f910f1fbc86a30c. Latest consensus state: current_term: 1 leader_uuid: "7a39333fc64049a59f910f1fbc86a30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a39333fc64049a59f910f1fbc86a30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34729 } } }
I20260619 20:54:01.226464 21476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:01.226478 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:01.226749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21404
I20260619 20:54:01.226821 21479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:01.226873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:01.227041 392 external_mini_cluster.cc:1428] Setting key 59af0c8e786d5c2fbabde5f771ccfdab
I20260619 20:54:01.227347 21479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:01.228904 21479 catalog_manager.cc:1347] Generated new cluster ID: ce8c4a3a49084cc182114cb1175e5c64
I20260619 20:54:01.228964 21479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:01.232353 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:01.227686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:41527 (local address 127.0.0.1:34729)
0619 20:54:01.227838 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:01.227848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:01.227878 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:54:01.227989 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:01.227996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:01.228114 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:01.228274 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:01.228601 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:01.229711 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:01.230509 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:01.230935 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:01.231051 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:01.231774 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:01.231777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:01.231778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:01.231954 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:01.231983 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:01.231993 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:01.232084 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:01.232161 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:54:01.232189 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260619 20:54:01.233256 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:01.237699 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:01.234786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:59369 (local address 127.0.0.1:34729)
0619 20:54:01.234843 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:01.234845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:01.235025 (+ 180us) server_negotiation.cc:407] Connection header received
0619 20:54:01.235095 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:01.235096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:01.235107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:01.235161 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:01.235291 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:01.235763 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:01.236495 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:01.236706 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:01.236780 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:01.237449 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:01.237451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:01.237451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:01.237455 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:01.237469 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:01.237474 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:01.237524 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:01.237618 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 20:54:01.237631 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:54:01.237879 21479 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:01.238538 21479 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:01.245910 21479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a39333fc64049a59f910f1fbc86a30c: Generated new TSK 0
I20260619 20:54:01.246125 21479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:01.251883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21404
W20260619 20:54:01.256407 21389 connection.cc:535] client connection to 127.0.0.1:34729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34729): Connection reset by peer (error 104)
2026-06-19T20:54:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:01 dist-test-slave-0smx krb5kdc[21503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:01 dist-test-slave-0smx krb5kdc[21503](info): set up 2 sockets
Jun 19 20:54:01 dist-test-slave-0smx krb5kdc[21503](info): commencing operation
krb5kdc: starting...
W20260619 20:54:02.285285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:02 dist-test-slave-0smx krb5kdc[21503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:54:02.401871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:02.481005 21516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:02.481225 21516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:02.481247 21516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:02.482419 21516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:02.482460 21516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:02.482475 21516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:02.482487 21516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:02.482542 21516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:02.483991 21516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:02.484226 21516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:02.484467 21516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:02.489149 21524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:02.489233 21516 server_base.cc:1034] running on GCE node
W20260619 20:54:02.489249 21521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:02.489149 21522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:02.489707 21516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:02.490032 21516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:02.491171 21516 hybrid_clock.cc:648] HybridClock initialized: now 1781902442491159 us; error 46 us; skew 500 ppm
Jun 19 20:54:02 dist-test-slave-0smx krb5kdc[21503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:02.493623 21516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:54:02.494060 21516 webserver.cc:460] Webserver started at http://127.0.0.1:36801/ using document root <none> and password file <none>
I20260619 20:54:02.494288 21516 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:02.494354 21516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:02.494506 21516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:02.495544 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "fe36ec1f9cdc40e0bde80cc8957befd7"
format_stamp: "Formatted at 2026-06-19 20:54:02 on dist-test-slave-0smx"
server_key: "6a60e37aaef2b8b1fa58647be174969a"
server_key_iv: "d45df75a6726dcb72dc94044f21e2857"
server_key_version: "encryptionkey@0"
I20260619 20:54:02.495898 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe36ec1f9cdc40e0bde80cc8957befd7"
format_stamp: "Formatted at 2026-06-19 20:54:02 on dist-test-slave-0smx"
server_key: "6a60e37aaef2b8b1fa58647be174969a"
server_key_iv: "d45df75a6726dcb72dc94044f21e2857"
server_key_version: "encryptionkey@0"
I20260619 20:54:02.497182 21516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:54:02.497965 21531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:02.498157 21516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:54:02.498265 21516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "fe36ec1f9cdc40e0bde80cc8957befd7"
format_stamp: "Formatted at 2026-06-19 20:54:02 on dist-test-slave-0smx"
server_key: "6a60e37aaef2b8b1fa58647be174969a"
server_key_iv: "d45df75a6726dcb72dc94044f21e2857"
server_key_version: "encryptionkey@0"
I20260619 20:54:02.498343 21516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:02.501583 21516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:02.503435 21516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:02.503592 21516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:02.507602 21516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33889
I20260619 20:54:02.507633 21583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33889 every 8 connection(s)
I20260619 20:54:02.507939 21516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:02.508824 21584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:02.511117 21584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7: Bootstrap starting.
I20260619 20:54:02.511679 21584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:02.511955 21584 log.cc:824] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:02.512629 21584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7: No bootstrap required, opened a new log
I20260619 20:54:02.514307 21584 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33889 } }
I20260619 20:54:02.514437 21584 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:02.514459 21584 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36ec1f9cdc40e0bde80cc8957befd7, State: Initialized, Role: FOLLOWER
I20260619 20:54:02.514614 21584 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33889 } }
I20260619 20:54:02.514672 21584 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:02.514729 21584 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:02.514779 21584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:02.515305 21584 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33889 } }
I20260619 20:54:02.515389 21584 leader_election.cc:302] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe36ec1f9cdc40e0bde80cc8957befd7; no voters:
I20260619 20:54:02.515555 21584 leader_election.cc:288] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:02.515606 21587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:02.515837 21587 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [term 1 LEADER]: Becoming Leader. State: Replica: fe36ec1f9cdc40e0bde80cc8957befd7, State: Running, Role: LEADER
I20260619 20:54:02.515836 21584 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:02.515955 21587 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33889 } }
I20260619 20:54:02.516168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21516
I20260619 20:54:02.516304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:02.516484 392 external_mini_cluster.cc:1428] Setting key 404ac95084d8929bd0724e51cb5ebcb0
I20260619 20:54:02.516722 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe36ec1f9cdc40e0bde80cc8957befd7. Latest consensus state: current_term: 1 leader_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33889 } } }
I20260619 20:54:02.516794 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:02.517015 21596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:02.517458 21588 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe36ec1f9cdc40e0bde80cc8957befd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33889 } } }
I20260619 20:54:02.517637 21588 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:02.517542 21596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:02.519829 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:02.516974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:32899 (local address 127.0.0.1:33889)
0619 20:54:02.518285 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:02.518294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:02.518327 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:54:02.518437 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:02.518443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:02.518474 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:02.518623 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:02.519362 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:02.519366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:02.519366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:02.519371 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:02.519410 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:02.519424 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:02.519497 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:02.519615 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 20:54:02.519663 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":66,"threads_started":1}
I20260619 20:54:02.520011 21596 catalog_manager.cc:1347] Generated new cluster ID: 6b68be9702a54f0f91250314ee892415
I20260619 20:54:02.520059 21596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:02.520924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:02.523425 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:02.522348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:59943 (local address 127.0.0.1:33889)
0619 20:54:02.522381 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:02.522383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:02.522484 (+ 101us) server_negotiation.cc:407] Connection header received
0619 20:54:02.522567 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:02.522567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:02.522572 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:02.522608 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:02.523221 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:02.523222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:02.523223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:02.523226 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:02.523239 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:02.523243 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:02.523270 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:02.523355 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:54:02.523368 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:54:02.527456 21596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:02.527626 21596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:02.534487 21596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe36ec1f9cdc40e0bde80cc8957befd7: Generated new TSK 0
I20260619 20:54:02.534674 21596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:02.537554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21516
2026-06-19T20:54:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:02 dist-test-slave-0smx krb5kdc[21615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:02 dist-test-slave-0smx krb5kdc[21615](info): set up 2 sockets
Jun 19 20:54:02 dist-test-slave-0smx krb5kdc[21615](info): commencing operation
krb5kdc: starting...
W20260619 20:54:03.573460 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:03 dist-test-slave-0smx krb5kdc[21615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:54:03.685778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:03.764940 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:03.765157 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:03.765180 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:03.766252 21628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:03.766287 21628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:03.766301 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:03.766314 21628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:03.766325 21628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:03.767690 21628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:03.767869 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:03.768047 21628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:03.772883 21634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:03.772897 21636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:03.772866 21633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:03.773062 21628 server_base.cc:1034] running on GCE node
I20260619 20:54:03.773420 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:03.773707 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:03.774849 21628 hybrid_clock.cc:648] HybridClock initialized: now 1781902443774840 us; error 33 us; skew 500 ppm
Jun 19 20:54:03 dist-test-slave-0smx krb5kdc[21615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:03.777417 21628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:54:03.777887 21628 webserver.cc:460] Webserver started at http://127.0.0.1:35805/ using document root <none> and password file <none>
I20260619 20:54:03.778120 21628 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:03.778189 21628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:03.778344 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:03.779400 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "48066556a1f24b118571031e9437a326"
format_stamp: "Formatted at 2026-06-19 20:54:03 on dist-test-slave-0smx"
server_key: "f947d913bc4fe9420d550f25655c3342"
server_key_iv: "718e7da971cd3e4aaf54d93fe6caeac3"
server_key_version: "encryptionkey@0"
I20260619 20:54:03.779764 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "48066556a1f24b118571031e9437a326"
format_stamp: "Formatted at 2026-06-19 20:54:03 on dist-test-slave-0smx"
server_key: "f947d913bc4fe9420d550f25655c3342"
server_key_iv: "718e7da971cd3e4aaf54d93fe6caeac3"
server_key_version: "encryptionkey@0"
I20260619 20:54:03.781067 21628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:03.781862 21643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:03.782053 21628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:03.782151 21628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "48066556a1f24b118571031e9437a326"
format_stamp: "Formatted at 2026-06-19 20:54:03 on dist-test-slave-0smx"
server_key: "f947d913bc4fe9420d550f25655c3342"
server_key_iv: "718e7da971cd3e4aaf54d93fe6caeac3"
server_key_version: "encryptionkey@0"
I20260619 20:54:03.782238 21628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:03.802642 21628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:03.805030 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:03.805205 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:03.809574 21628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46595
I20260619 20:54:03.809608 21695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46595 every 8 connection(s)
I20260619 20:54:03.809975 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:03.810835 21696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:03.813004 21696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326: Bootstrap starting.
I20260619 20:54:03.813531 21696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:03.813762 21696 log.cc:824] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:03.814390 21696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326: No bootstrap required, opened a new log
I20260619 20:54:03.815914 21696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48066556a1f24b118571031e9437a326" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260619 20:54:03.816047 21696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:03.816068 21696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48066556a1f24b118571031e9437a326, State: Initialized, Role: FOLLOWER
I20260619 20:54:03.816205 21696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48066556a1f24b118571031e9437a326" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260619 20:54:03.816289 21696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:03.816313 21696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:03.816340 21696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:03.816809 21696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48066556a1f24b118571031e9437a326" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260619 20:54:03.816891 21696 leader_election.cc:302] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48066556a1f24b118571031e9437a326; no voters:
I20260619 20:54:03.817025 21696 leader_election.cc:288] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:03.817090 21699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:03.817274 21699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [term 1 LEADER]: Becoming Leader. State: Replica: 48066556a1f24b118571031e9437a326, State: Running, Role: LEADER
I20260619 20:54:03.817286 21696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:03.817423 21699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48066556a1f24b118571031e9437a326" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260619 20:54:03.817878 21701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48066556a1f24b118571031e9437a326. Latest consensus state: current_term: 1 leader_uuid: "48066556a1f24b118571031e9437a326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48066556a1f24b118571031e9437a326" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260619 20:54:03.817895 21700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48066556a1f24b118571031e9437a326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48066556a1f24b118571031e9437a326" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260619 20:54:03.818044 21701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:03.818130 21700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:03.818387 21705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:03.818940 21705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:03.820127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21628
I20260619 20:54:03.820271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:03.820487 392 external_mini_cluster.cc:1428] Setting key d36df3399665c368277f250f4f761968
I20260619 20:54:03.821074 21705 catalog_manager.cc:1347] Generated new cluster ID: 400d719de5614aad8fcf670d491f9f95
I20260619 20:54:03.821122 21705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:54:03 dist-test-slave-0smx krb5kdc[21615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:54:03.832540 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:03.821087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:55195 (local address 127.0.0.1:46595)
0619 20:54:03.821236 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:03.821245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:03.821276 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:54:03.821403 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:03.821409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:03.821463 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:03.821592 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:03.822382 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:03.823781 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:03.824860 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:03.825884 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:03.825988 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:03.828849 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:03.828854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:03.828854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:03.828866 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:03.830781 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:03.831238 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:03.831241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:03.831241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:03.831337 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:03.831567 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:03.831567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:03.831568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:03.831841 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:03.831972 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:03.832097 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 20:54:03.832283 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260619 20:54:03.833539 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:03.841809 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:03.834929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:47361 (local address 127.0.0.1:46595)
0619 20:54:03.834966 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:03.834968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:03.835069 (+ 101us) server_negotiation.cc:407] Connection header received
0619 20:54:03.835146 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:03.835146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:03.835154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:03.835178 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:03.835797 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:03.836537 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:03.837522 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:03.837828 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:03.837864 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:03.839200 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:03.839201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:03.839202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:03.839208 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:03.840630 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:03.841010 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:03.841011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:03.841012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:03.841056 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:03.841250 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:03.841251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:03.841251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:03.841385 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:03.841464 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:03.841663 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 20:54:03.841722 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:54:03.854902 21705 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:03.855635 21705 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:03.860746 21705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48066556a1f24b118571031e9437a326: Generated new TSK 0
I20260619 20:54:03.860930 21705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:03.863997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21628
W20260619 20:54:03.867940 21613 connection.cc:535] client connection to 127.0.0.1:46595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46595): Connection reset by peer (error 104)
2026-06-19T20:54:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:03 dist-test-slave-0smx krb5kdc[21727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:03 dist-test-slave-0smx krb5kdc[21727](info): set up 2 sockets
Jun 19 20:54:03 dist-test-slave-0smx krb5kdc[21727](info): commencing operation
krb5kdc: starting...
W20260619 20:54:04.893599 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:04 dist-test-slave-0smx krb5kdc[21727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:54:05.011548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:05.091897 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:05.092144 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:05.092181 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:05.093343 21740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:05.093411 21740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:05.093434 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:05.093456 21740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:05.093483 21740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:05.094930 21740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:05.095212 21740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:05.095440 21740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:05.100359 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:05.100528 21740 server_base.cc:1034] running on GCE node
W20260619 20:54:05.100350 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:05.100350 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:05.100883 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:05.101166 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:05.102317 21740 hybrid_clock.cc:648] HybridClock initialized: now 1781902445102301 us; error 27 us; skew 500 ppm
Jun 19 20:54:05 dist-test-slave-0smx krb5kdc[21727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:05.109227 21740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:54:05.109740 21740 webserver.cc:460] Webserver started at http://127.0.0.1:37779/ using document root <none> and password file <none>
I20260619 20:54:05.109970 21740 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:05.110037 21740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:05.110188 21740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:05.111248 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "7fa21ca032e048839d61ccf457ba7381"
format_stamp: "Formatted at 2026-06-19 20:54:05 on dist-test-slave-0smx"
server_key: "e6ffeadb54d9e74dbe607b70e13b3a26"
server_key_iv: "22a6a0f7c093e3e0bce488cc6f1cc762"
server_key_version: "encryptionkey@0"
I20260619 20:54:05.111604 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fa21ca032e048839d61ccf457ba7381"
format_stamp: "Formatted at 2026-06-19 20:54:05 on dist-test-slave-0smx"
server_key: "e6ffeadb54d9e74dbe607b70e13b3a26"
server_key_iv: "22a6a0f7c093e3e0bce488cc6f1cc762"
server_key_version: "encryptionkey@0"
I20260619 20:54:05.112901 21740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:05.113682 21755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:05.113870 21740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:05.113974 21740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "7fa21ca032e048839d61ccf457ba7381"
format_stamp: "Formatted at 2026-06-19 20:54:05 on dist-test-slave-0smx"
server_key: "e6ffeadb54d9e74dbe607b70e13b3a26"
server_key_iv: "22a6a0f7c093e3e0bce488cc6f1cc762"
server_key_version: "encryptionkey@0"
I20260619 20:54:05.114046 21740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:05.122081 21740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:05.123842 21740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:05.123992 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:05.128134 21740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35525
I20260619 20:54:05.128168 21807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35525 every 8 connection(s)
I20260619 20:54:05.128504 21740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:05.129488 21808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:05.131676 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381: Bootstrap starting.
I20260619 20:54:05.132226 21808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:05.132499 21808 log.cc:824] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:05.133150 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381: No bootstrap required, opened a new log
I20260619 20:54:05.134721 21808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa21ca032e048839d61ccf457ba7381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260619 20:54:05.134850 21808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:05.134872 21808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa21ca032e048839d61ccf457ba7381, State: Initialized, Role: FOLLOWER
I20260619 20:54:05.135046 21808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa21ca032e048839d61ccf457ba7381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260619 20:54:05.135131 21808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:05.135183 21808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:05.135233 21808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:05.135726 21808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa21ca032e048839d61ccf457ba7381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260619 20:54:05.135831 21808 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa21ca032e048839d61ccf457ba7381; no voters:
I20260619 20:54:05.136008 21808 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:05.136026 21811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:05.136106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21740
I20260619 20:54:05.136143 21811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa21ca032e048839d61ccf457ba7381, State: Running, Role: LEADER
I20260619 20:54:05.136220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:05.136214 21811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa21ca032e048839d61ccf457ba7381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260619 20:54:05.136257 21808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:05.136407 392 external_mini_cluster.cc:1428] Setting key ccd5c0f17ef3cd67944a515acb11100c
I20260619 20:54:05.136533 21811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa21ca032e048839d61ccf457ba7381. Latest consensus state: current_term: 1 leader_uuid: "7fa21ca032e048839d61ccf457ba7381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa21ca032e048839d61ccf457ba7381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260619 20:54:05.136590 21811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:05.136694 21812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa21ca032e048839d61ccf457ba7381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa21ca032e048839d61ccf457ba7381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260619 20:54:05.136773 21812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:05.137459 21816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:05.137955 21816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:05.141752 21816 catalog_manager.cc:1347] Generated new cluster ID: 0e1c9f3cde4e49048fcb414027f8b2b2
I20260619 20:54:05.141799 21816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:05.146384 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:05.137059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:38671 (local address 127.0.0.1:35525)
0619 20:54:05.137807 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:05.137816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:05.137847 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:54:05.137944 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:05.137949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:05.137997 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:05.138144 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:05.138397 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:05.139754 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:05.140533 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:05.141160 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:05.141270 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:05.145580 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:05.145584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:05.145584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:05.145793 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:05.145830 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:05.145844 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:05.145921 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:05.146152 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 20:54:05.146178 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":77,"threads_started":1}
I20260619 20:54:05.147253 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:05.153172 21816 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:05.153730 21816 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:05.156553 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:05.149649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:41077 (local address 127.0.0.1:35525)
0619 20:54:05.149690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:05.149692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:05.153033 (+ 3341us) server_negotiation.cc:407] Connection header received
0619 20:54:05.153056 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:05.153057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:05.153068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:05.153164 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:05.153417 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:05.154205 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:05.155223 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:05.155548 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:05.155621 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:05.156310 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:05.156312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:05.156312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:05.156316 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:05.156330 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:05.156334 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:05.156396 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:05.156466 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:54:05.156479 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:54:05.162487 21816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa21ca032e048839d61ccf457ba7381: Generated new TSK 0
I20260619 20:54:05.162658 21816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:05.168857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21740
W20260619 20:54:05.172731 21725 connection.cc:535] client connection to 127.0.0.1:35525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35525): Connection reset by peer (error 104)
2026-06-19T20:54:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:05 dist-test-slave-0smx krb5kdc[21839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:05 dist-test-slave-0smx krb5kdc[21839](info): set up 2 sockets
Jun 19 20:54:05 dist-test-slave-0smx krb5kdc[21839](info): commencing operation
krb5kdc: starting...
W20260619 20:54:06.189496 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:06 dist-test-slave-0smx krb5kdc[21839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:54:06.307312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:06.386633 21852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:06.386837 21852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:06.386857 21852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:06.387943 21852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:06.387989 21852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:06.388038 21852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:06.388065 21852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:06.388092 21852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:06.389554 21852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:06.389782 21852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:06.390048 21852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:06.394780 21857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:06.394773 21860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:06.394937 21852 server_base.cc:1034] running on GCE node
W20260619 20:54:06.394776 21858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:06.395280 21852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:06.395552 21852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:06.396716 21852 hybrid_clock.cc:648] HybridClock initialized: now 1781902446396685 us; error 28 us; skew 500 ppm
Jun 19 20:54:06 dist-test-slave-0smx krb5kdc[21839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:06.399077 21852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:54:06.399631 21852 webserver.cc:460] Webserver started at http://127.0.0.1:45961/ using document root <none> and password file <none>
I20260619 20:54:06.399963 21852 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:06.400031 21852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:06.400163 21852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:06.401577 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "ccbd97cdc5974993a344c66a62adf06e"
format_stamp: "Formatted at 2026-06-19 20:54:06 on dist-test-slave-0smx"
server_key: "2fe7d20c083a97a8265d1b7e683f581d"
server_key_iv: "3093b2f80abb46b8cf979d56e9303319"
server_key_version: "encryptionkey@0"
I20260619 20:54:06.401937 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccbd97cdc5974993a344c66a62adf06e"
format_stamp: "Formatted at 2026-06-19 20:54:06 on dist-test-slave-0smx"
server_key: "2fe7d20c083a97a8265d1b7e683f581d"
server_key_iv: "3093b2f80abb46b8cf979d56e9303319"
server_key_version: "encryptionkey@0"
I20260619 20:54:06.403342 21852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:54:06.404059 21867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:06.404218 21852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:06.404315 21852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "ccbd97cdc5974993a344c66a62adf06e"
format_stamp: "Formatted at 2026-06-19 20:54:06 on dist-test-slave-0smx"
server_key: "2fe7d20c083a97a8265d1b7e683f581d"
server_key_iv: "3093b2f80abb46b8cf979d56e9303319"
server_key_version: "encryptionkey@0"
I20260619 20:54:06.404393 21852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:06.407536 21852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:06.409262 21852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:06.409427 21852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:06.413146 21852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34519
I20260619 20:54:06.413177 21919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34519 every 8 connection(s)
I20260619 20:54:06.413514 21852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:06.414490 21920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:06.416586 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e: Bootstrap starting.
I20260619 20:54:06.417130 21920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:06.417474 21920 log.cc:824] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:06.418102 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e: No bootstrap required, opened a new log
I20260619 20:54:06.419595 21920 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbd97cdc5974993a344c66a62adf06e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260619 20:54:06.419716 21920 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:06.419763 21920 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbd97cdc5974993a344c66a62adf06e, State: Initialized, Role: FOLLOWER
I20260619 20:54:06.419938 21920 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbd97cdc5974993a344c66a62adf06e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260619 20:54:06.420034 21920 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:06.420074 21920 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:06.420122 21920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:06.420627 21920 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbd97cdc5974993a344c66a62adf06e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260619 20:54:06.420738 21920 leader_election.cc:302] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbd97cdc5974993a344c66a62adf06e; no voters:
I20260619 20:54:06.420914 21920 leader_election.cc:288] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:06.421018 21923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:06.421157 21920 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:06.421479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21852
I20260619 20:54:06.421162 21923 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [term 1 LEADER]: Becoming Leader. State: Replica: ccbd97cdc5974993a344c66a62adf06e, State: Running, Role: LEADER
I20260619 20:54:06.421590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:06.421681 21923 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbd97cdc5974993a344c66a62adf06e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260619 20:54:06.421767 392 external_mini_cluster.cc:1428] Setting key 05cdf8262210bd820c77315442157237
I20260619 20:54:06.422142 21924 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbd97cdc5974993a344c66a62adf06e. Latest consensus state: current_term: 1 leader_uuid: "ccbd97cdc5974993a344c66a62adf06e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbd97cdc5974993a344c66a62adf06e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260619 20:54:06.422257 21924 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:06.422701 21925 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccbd97cdc5974993a344c66a62adf06e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbd97cdc5974993a344c66a62adf06e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260619 20:54:06.422824 21925 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:06.422999 21932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:06.423568 21932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:06.424319 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:06.422385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:34101 (local address 127.0.0.1:34519)
0619 20:54:06.422569 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:06.422577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:06.422709 (+ 132us) server_negotiation.cc:407] Connection header received
0619 20:54:06.422840 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:06.422847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:06.422878 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:06.423286 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:06.423753 (+ 467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:06.423757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:06.423758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:06.423762 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:06.423811 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:06.423828 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:06.423886 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:06.424057 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 20:54:06.424104 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260619 20:54:06.425354 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:06.426707 21932 catalog_manager.cc:1347] Generated new cluster ID: 3ba116436b57401084a481f350bf491f
I20260619 20:54:06.426841 21932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:06.434651 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:06.426808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:47071 (local address 127.0.0.1:34519)
0619 20:54:06.426855 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:06.426857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:06.433623 (+ 6766us) server_negotiation.cc:407] Connection header received
0619 20:54:06.433710 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:06.433711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:06.433716 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:06.433789 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:06.434430 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:06.434431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:06.434432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:06.434436 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:06.434450 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:06.434454 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:06.434519 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:06.434573 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 20:54:06.434587 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:54:06.461274 21932 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:06.461508 21932 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:06.464363 21932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccbd97cdc5974993a344c66a62adf06e: Generated new TSK 0
I20260619 20:54:06.464558 21932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:06.466128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21852
2026-06-19T20:54:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:06 dist-test-slave-0smx krb5kdc[21951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:06 dist-test-slave-0smx krb5kdc[21951](info): set up 2 sockets
Jun 19 20:54:06 dist-test-slave-0smx krb5kdc[21951](info): commencing operation
krb5kdc: starting...
W20260619 20:54:07.497876 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:07 dist-test-slave-0smx krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:07.607972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37341
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:07.686592 21964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:07.686815 21964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:07.686849 21964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:07.687924 21964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:07.687987 21964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:07.688011 21964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:07.688038 21964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:07.688060 21964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:07.689817 21964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:07.690080 21964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:07.690287 21964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:07.694947 21969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:07.694955 21970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:07.695086 21964 server_base.cc:1034] running on GCE node
W20260619 20:54:07.694958 21972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:07.695451 21964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:07.695725 21964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:07.696859 21964 hybrid_clock.cc:648] HybridClock initialized: now 1781902447696852 us; error 28 us; skew 500 ppm
Jun 19 20:54:07 dist-test-slave-0smx krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:07.699219 21964 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:07.699570 21964 webserver.cc:460] Webserver started at http://127.0.98.62:41001/ using document root <none> and password file <none>
I20260619 20:54:07.699734 21964 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:07.699771 21964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:07.699858 21964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:07.700855 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "a95b45845e6543ff81fa6d53c4ffec68"
format_stamp: "Formatted at 2026-06-19 20:54:07 on dist-test-slave-0smx"
server_key: "8d4323bb43bb91d576b46c556f7b2647"
server_key_iv: "b815181717d476dd1a136380004e3645"
server_key_version: "encryptionkey@0"
I20260619 20:54:07.701154 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "a95b45845e6543ff81fa6d53c4ffec68"
format_stamp: "Formatted at 2026-06-19 20:54:07 on dist-test-slave-0smx"
server_key: "8d4323bb43bb91d576b46c556f7b2647"
server_key_iv: "b815181717d476dd1a136380004e3645"
server_key_version: "encryptionkey@0"
I20260619 20:54:07.702440 21964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:54:07.703056 21979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:07.703212 21964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:07.703292 21964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "a95b45845e6543ff81fa6d53c4ffec68"
format_stamp: "Formatted at 2026-06-19 20:54:07 on dist-test-slave-0smx"
server_key: "8d4323bb43bb91d576b46c556f7b2647"
server_key_iv: "b815181717d476dd1a136380004e3645"
server_key_version: "encryptionkey@0"
I20260619 20:54:07.703344 21964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:07.722505 21964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:07.724395 21964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:07.724494 21964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:07.728161 21964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37341
I20260619 20:54:07.728257 22031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37341 every 8 connection(s)
I20260619 20:54:07.728533 21964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:07.729393 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:07.731631 22032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68: Bootstrap starting.
I20260619 20:54:07.732347 22032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:07.732362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 21964
I20260619 20:54:07.732472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:07.732684 392 external_mini_cluster.cc:1428] Setting key a76909916991bbff5c9e467f45510c6d
I20260619 20:54:07.732697 22032 log.cc:824] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:07.733511 22032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68: No bootstrap required, opened a new log
I20260619 20:54:07.735993 22032 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b45845e6543ff81fa6d53c4ffec68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37341 } }
I20260619 20:54:07.736167 22032 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:07.736203 22032 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95b45845e6543ff81fa6d53c4ffec68, State: Initialized, Role: FOLLOWER
I20260619 20:54:07.736313 22032 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b45845e6543ff81fa6d53c4ffec68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37341 } }
I20260619 20:54:07.736380 22032 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:07.736406 22032 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:07.736449 22032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:07.737036 22032 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b45845e6543ff81fa6d53c4ffec68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37341 } }
I20260619 20:54:07.737126 22032 leader_election.cc:302] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95b45845e6543ff81fa6d53c4ffec68; no voters:
I20260619 20:54:07.737360 22032 leader_election.cc:288] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:07.737380 22037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:07.737517 22037 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [term 1 LEADER]: Becoming Leader. State: Replica: a95b45845e6543ff81fa6d53c4ffec68, State: Running, Role: LEADER
I20260619 20:54:07.737646 22037 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b45845e6543ff81fa6d53c4ffec68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37341 } }
I20260619 20:54:07.737900 22032 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:07.738034 22037 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95b45845e6543ff81fa6d53c4ffec68. Latest consensus state: current_term: 1 leader_uuid: "a95b45845e6543ff81fa6d53c4ffec68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b45845e6543ff81fa6d53c4ffec68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37341 } } }
I20260619 20:54:07.738116 22037 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:07.738045 22038 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95b45845e6543ff81fa6d53c4ffec68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b45845e6543ff81fa6d53c4ffec68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37341 } } }
I20260619 20:54:07.738188 22038 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:07.738996 22045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:07.739460 22045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:54:07 dist-test-slave-0smx krb5kdc[21951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:07.742333 22045 catalog_manager.cc:1347] Generated new cluster ID: 1b0464d421994196bf1f74e7864f029f
I20260619 20:54:07.742383 22045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:07.743848 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:07.733169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43037 (local address 127.0.98.62:37341)
0619 20:54:07.733375 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:07.733383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:07.733418 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:54:07.733921 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:07.733928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:07.733973 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:07.734081 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:07.734855 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:07.735708 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:07.736786 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:07.737220 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:07.737341 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:07.740338 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:07.740342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:07.740342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:07.740370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:07.742175 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:07.742608 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:07.742611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:07.742612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:07.742718 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:07.742955 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:07.742956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:07.742956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:07.743254 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:07.743422 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:07.743511 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:54:07.743624 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260619 20:54:07.744793 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:07.757200 22045 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:07.757932 22045 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:07.761925 22045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a95b45845e6543ff81fa6d53c4ffec68: Generated new TSK 0
I20260619 20:54:07.762086 22045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:07.763708 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:07.746124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58029 (local address 127.0.98.62:37341)
0619 20:54:07.746156 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:07.746158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:07.756394 (+ 10236us) server_negotiation.cc:407] Connection header received
0619 20:54:07.756484 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:07.756485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:07.756496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:07.756551 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:07.757178 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:07.758253 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:07.759235 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:07.759564 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:07.759617 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:07.761047 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:07.761049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:07.761049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:07.761055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:07.762672 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:07.763050 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:07.763050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:07.763051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:07.763097 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:07.763303 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:07.763303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:07.763303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:07.763388 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:07.763467 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:07.763608 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 20:54:07.763652 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:54:07.765525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 21964
W20260619 20:54:07.769232 21949 connection.cc:535] client connection to 127.0.98.62:37341 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37341): Connection reset by peer (error 104)
2026-06-19T20:54:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:07 dist-test-slave-0smx krb5kdc[22063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:07 dist-test-slave-0smx krb5kdc[22063](info): set up 2 sockets
Jun 19 20:54:07 dist-test-slave-0smx krb5kdc[22063](info): commencing operation
krb5kdc: starting...
W20260619 20:54:08.813483 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:08 dist-test-slave-0smx krb5kdc[22063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:08.926334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36181
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:09.005239 22076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:09.005493 22076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:09.005530 22076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:09.006697 22076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:09.006765 22076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:09.006789 22076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:09.006819 22076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:09.006871 22076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:09.008327 22076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:09.008601 22076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:09.008819 22076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:09.013600 22081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:09.013614 22084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:09.013644 22076 server_base.cc:1034] running on GCE node
W20260619 20:54:09.013664 22082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:09.014083 22076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:09.014385 22076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:09.015579 22076 hybrid_clock.cc:648] HybridClock initialized: now 1781902449015571 us; error 35 us; skew 500 ppm
Jun 19 20:54:09 dist-test-slave-0smx krb5kdc[22063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:09.018205 22076 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:09.018654 22076 webserver.cc:460] Webserver started at http://127.0.98.62:46673/ using document root <none> and password file <none>
I20260619 20:54:09.018889 22076 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:09.018963 22076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:09.019131 22076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:09.020269 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "86f277d0f251497b95ba536c8ac6f073"
format_stamp: "Formatted at 2026-06-19 20:54:09 on dist-test-slave-0smx"
server_key: "fa59bb91326c116d1b8f4058096c0e20"
server_key_iv: "0c39ed078da9bcae36bae60956ac821e"
server_key_version: "encryptionkey@0"
I20260619 20:54:09.020660 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "86f277d0f251497b95ba536c8ac6f073"
format_stamp: "Formatted at 2026-06-19 20:54:09 on dist-test-slave-0smx"
server_key: "fa59bb91326c116d1b8f4058096c0e20"
server_key_iv: "0c39ed078da9bcae36bae60956ac821e"
server_key_version: "encryptionkey@0"
I20260619 20:54:09.022130 22076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:09.022889 22091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:09.023056 22076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:09.023164 22076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "86f277d0f251497b95ba536c8ac6f073"
format_stamp: "Formatted at 2026-06-19 20:54:09 on dist-test-slave-0smx"
server_key: "fa59bb91326c116d1b8f4058096c0e20"
server_key_iv: "0c39ed078da9bcae36bae60956ac821e"
server_key_version: "encryptionkey@0"
I20260619 20:54:09.023249 22076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:09.031584 22076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:09.033478 22076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:09.033646 22076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:09.037786 22076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36181
I20260619 20:54:09.037812 22143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36181 every 8 connection(s)
I20260619 20:54:09.038177 22076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:09.039062 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:09.040679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22076
I20260619 20:54:09.040856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:09.041052 392 external_mini_cluster.cc:1428] Setting key d07391bb18463b4731a56a722346240a
I20260619 20:54:09.041596 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073: Bootstrap starting.
I20260619 20:54:09.042290 22144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:09.042613 22144 log.cc:824] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:09.043368 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073: No bootstrap required, opened a new log
I20260619 20:54:09.045732 22144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f277d0f251497b95ba536c8ac6f073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260619 20:54:09.045921 22144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:09.045962 22144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f277d0f251497b95ba536c8ac6f073, State: Initialized, Role: FOLLOWER
I20260619 20:54:09.046092 22144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f277d0f251497b95ba536c8ac6f073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260619 20:54:09.046186 22144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:09.046218 22144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:09.046275 22144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:09.046828 22144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f277d0f251497b95ba536c8ac6f073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260619 20:54:09.046986 22144 leader_election.cc:302] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f277d0f251497b95ba536c8ac6f073; no voters:
I20260619 20:54:09.047139 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:09.041581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38415 (local address 127.0.98.62:36181)
0619 20:54:09.041770 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:09.041779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:09.041811 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:54:09.042446 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:09.042457 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:09.042522 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:09.042673 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:09.042957 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:09.044007 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:09.045137 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:09.045583 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:09.045662 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:09.046476 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:09.046481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:09.046481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:09.046651 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:09.046681 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:09.046692 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:09.046780 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:09.046860 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 20:54:09.046887 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260619 20:54:09.047186 22144 leader_election.cc:288] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:09.047237 22149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:09.047394 22149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [term 1 LEADER]: Becoming Leader. State: Replica: 86f277d0f251497b95ba536c8ac6f073, State: Running, Role: LEADER
I20260619 20:54:09.047451 22144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:09.047662 22149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f277d0f251497b95ba536c8ac6f073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } }
I20260619 20:54:09.048097 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:09.048199 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f277d0f251497b95ba536c8ac6f073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f277d0f251497b95ba536c8ac6f073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } } }
I20260619 20:54:09.048269 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:09.048467 22158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:09.049054 22158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:09.049599 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f277d0f251497b95ba536c8ac6f073. Latest consensus state: current_term: 1 leader_uuid: "86f277d0f251497b95ba536c8ac6f073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f277d0f251497b95ba536c8ac6f073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36181 } } }
I20260619 20:54:09.049686 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:09.053484 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:09.050321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34669 (local address 127.0.98.62:36181)
0619 20:54:09.050369 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:09.050370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:09.050682 (+ 312us) server_negotiation.cc:407] Connection header received
0619 20:54:09.050738 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:09.050739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:09.050747 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:09.050818 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:09.050980 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:09.051513 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:09.052218 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:09.052438 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:09.052511 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:09.053190 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:09.053191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:09.053192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:09.053196 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:09.053214 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:09.053218 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:09.053323 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:09.053382 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:54:09.053390 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:54:09.053591 22158 catalog_manager.cc:1347] Generated new cluster ID: b2314336cbe04cadb839be65ee2ebd2a
I20260619 20:54:09.053628 22158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:09.069651 22158 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:09.070230 22158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:09.078393 22158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f277d0f251497b95ba536c8ac6f073: Generated new TSK 0
I20260619 20:54:09.078634 22158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:09.091130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22076
W20260619 20:54:09.095575 22061 connection.cc:535] client connection to 127.0.98.62:36181 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36181): Connection reset by peer (error 104)
2026-06-19T20:54:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:09 dist-test-slave-0smx krb5kdc[22175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:09 dist-test-slave-0smx krb5kdc[22175](info): set up 2 sockets
Jun 19 20:54:09 dist-test-slave-0smx krb5kdc[22175](info): commencing operation
krb5kdc: starting...
W20260619 20:54:10.125211 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:10 dist-test-slave-0smx krb5kdc[22175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:10.237497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:10.316115 22188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:10.316345 22188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:10.316385 22188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:10.317516 22188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:10.317582 22188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:10.317615 22188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:10.317647 22188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:10.317672 22188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:10.319119 22188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:10.319381 22188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:10.319598 22188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:10.324397 22193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:10.324412 22194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:10.324424 22196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:10.324604 22188 server_base.cc:1034] running on GCE node
I20260619 20:54:10.324841 22188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:10.325119 22188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:10.326264 22188 hybrid_clock.cc:648] HybridClock initialized: now 1781902450326244 us; error 33 us; skew 500 ppm
Jun 19 20:54:10 dist-test-slave-0smx krb5kdc[22175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:10.331885 22188 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:10.332367 22188 webserver.cc:460] Webserver started at http://127.0.98.62:37237/ using document root <none> and password file <none>
I20260619 20:54:10.332587 22188 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:10.332655 22188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:10.332803 22188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:10.333906 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "40fae7d5e06849d0b522bc869cdeaa8d"
format_stamp: "Formatted at 2026-06-19 20:54:10 on dist-test-slave-0smx"
server_key: "532d32f0c08794cef65c4a23cab6a2e5"
server_key_iv: "95a4a32f017de2000405eca06bc854ff"
server_key_version: "encryptionkey@0"
I20260619 20:54:10.334261 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "40fae7d5e06849d0b522bc869cdeaa8d"
format_stamp: "Formatted at 2026-06-19 20:54:10 on dist-test-slave-0smx"
server_key: "532d32f0c08794cef65c4a23cab6a2e5"
server_key_iv: "95a4a32f017de2000405eca06bc854ff"
server_key_version: "encryptionkey@0"
I20260619 20:54:10.335553 22188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:54:10.336288 22203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:10.336462 22188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:10.336558 22188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "40fae7d5e06849d0b522bc869cdeaa8d"
format_stamp: "Formatted at 2026-06-19 20:54:10 on dist-test-slave-0smx"
server_key: "532d32f0c08794cef65c4a23cab6a2e5"
server_key_iv: "95a4a32f017de2000405eca06bc854ff"
server_key_version: "encryptionkey@0"
I20260619 20:54:10.336634 22188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:10.339901 22188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:10.341945 22188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:10.342105 22188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:10.345754 22188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40675
I20260619 20:54:10.345814 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40675 every 8 connection(s)
I20260619 20:54:10.346134 22188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:10.347071 22256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:10.349174 22256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d: Bootstrap starting.
I20260619 20:54:10.349747 22256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:10.350018 22256 log.cc:824] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:10.350677 22256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d: No bootstrap required, opened a new log
I20260619 20:54:10.351699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22188
I20260619 20:54:10.351814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:10.352020 392 external_mini_cluster.cc:1428] Setting key 790718daeaadbee4dc766009e09c88cf
I20260619 20:54:10.352674 22256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260619 20:54:10.352833 22256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:10.352877 22256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fae7d5e06849d0b522bc869cdeaa8d, State: Initialized, Role: FOLLOWER
I20260619 20:54:10.353055 22256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260619 20:54:10.353165 22256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:10.353223 22256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:10.353276 22256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:10.354176 22256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260619 20:54:10.354236 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:10.352539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33057 (local address 127.0.98.62:40675)
0619 20:54:10.352754 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:10.352764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:10.352796 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:54:10.352933 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:10.352940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:10.352971 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:10.353156 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:10.353814 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:10.353817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:10.353818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:10.353821 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:10.353862 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:10.353873 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:10.353943 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:10.354039 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 20:54:10.354081 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260619 20:54:10.354285 22256 leader_election.cc:302] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fae7d5e06849d0b522bc869cdeaa8d; no voters:
I20260619 20:54:10.354482 22256 leader_election.cc:288] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:10.354516 22261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:10.354673 22261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [term 1 LEADER]: Becoming Leader. State: Replica: 40fae7d5e06849d0b522bc869cdeaa8d, State: Running, Role: LEADER
I20260619 20:54:10.354735 22256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:10.354888 22261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260619 20:54:10.355162 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:10.356102 22262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fae7d5e06849d0b522bc869cdeaa8d. Latest consensus state: current_term: 1 leader_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } } }
I20260619 20:54:10.356211 22262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:10.356383 22263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fae7d5e06849d0b522bc869cdeaa8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } } }
I20260619 20:54:10.356477 22263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d [sys.catalog]: This master's current role is: LEADER
W20260619 20:54:10.357177 22277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:54:10.357235 22277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:54:10.357400 22279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:10.357539 22279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:10.357909 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:10.356731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34133 (local address 127.0.98.62:40675)
0619 20:54:10.356789 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:10.356792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:10.356852 (+ 60us) server_negotiation.cc:407] Connection header received
0619 20:54:10.356924 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:10.356925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:10.356929 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:10.356984 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:10.357676 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:10.357677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:10.357677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:10.357682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:10.357695 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:10.357699 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:10.357757 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:10.357813 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 20:54:10.357827 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:54:10.358992 22279 catalog_manager.cc:1347] Generated new cluster ID: 03b361d9007b41d6b5188785af89ec9e
I20260619 20:54:10.359059 22279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:10.371325 22279 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:10.371482 22279 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:10.373914 22279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40fae7d5e06849d0b522bc869cdeaa8d: Generated new TSK 0
I20260619 20:54:10.374063 22279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:10.377519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22188
2026-06-19T20:54:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:10 dist-test-slave-0smx krb5kdc[22287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:10 dist-test-slave-0smx krb5kdc[22287](info): set up 2 sockets
Jun 19 20:54:10 dist-test-slave-0smx krb5kdc[22287](info): commencing operation
krb5kdc: starting...
W20260619 20:54:11.397538 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:11 dist-test-slave-0smx krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:11.509989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45105
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:11.589263 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:11.589526 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:11.589564 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:11.590729 22300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:11.590798 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:11.590822 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:11.590843 22300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:11.590873 22300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:11.592316 22300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:11.592590 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:11.592813 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:11.597608 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:11.597589 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:11.597640 22300 server_base.cc:1034] running on GCE node
W20260619 20:54:11.597606 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:11.598076 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:11.598310 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:11.599473 22300 hybrid_clock.cc:648] HybridClock initialized: now 1781902451599469 us; error 35 us; skew 500 ppm
Jun 19 20:54:11 dist-test-slave-0smx krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:11.601914 22300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:11.602353 22300 webserver.cc:460] Webserver started at http://127.0.98.62:39981/ using document root <none> and password file <none>
I20260619 20:54:11.602577 22300 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:11.602643 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:11.602783 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:11.603785 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "b21b15ffbc7e45d3a9cbb20291a55025"
format_stamp: "Formatted at 2026-06-19 20:54:11 on dist-test-slave-0smx"
server_key: "396d414c399d1c4bce430411de8d4178"
server_key_iv: "7125bace2967af0c83fcb683605470ef"
server_key_version: "encryptionkey@0"
I20260619 20:54:11.604141 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "b21b15ffbc7e45d3a9cbb20291a55025"
format_stamp: "Formatted at 2026-06-19 20:54:11 on dist-test-slave-0smx"
server_key: "396d414c399d1c4bce430411de8d4178"
server_key_iv: "7125bace2967af0c83fcb683605470ef"
server_key_version: "encryptionkey@0"
I20260619 20:54:11.605472 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:54:11.606247 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:11.606421 22300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:54:11.606516 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "b21b15ffbc7e45d3a9cbb20291a55025"
format_stamp: "Formatted at 2026-06-19 20:54:11 on dist-test-slave-0smx"
server_key: "396d414c399d1c4bce430411de8d4178"
server_key_iv: "7125bace2967af0c83fcb683605470ef"
server_key_version: "encryptionkey@0"
I20260619 20:54:11.606597 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:11.616968 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:11.619189 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:11.619349 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:11.623258 22300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45105
I20260619 20:54:11.623292 22367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45105 every 8 connection(s)
I20260619 20:54:11.623613 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:11.623957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22300
I20260619 20:54:11.624095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:11.624285 392 external_mini_cluster.cc:1428] Setting key 13476b6613b73661e4692e3bf4a76b52
I20260619 20:54:11.624611 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:11.627913 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025: Bootstrap starting.
I20260619 20:54:11.628495 22368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:11.628801 22368 log.cc:824] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:11.629518 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025: No bootstrap required, opened a new log
Jun 19 20:54:11 dist-test-slave-0smx krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:11.631340 22368 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45105 } }
I20260619 20:54:11.631485 22368 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:11.631547 22368 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21b15ffbc7e45d3a9cbb20291a55025, State: Initialized, Role: FOLLOWER
I20260619 20:54:11.631696 22368 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45105 } }
I20260619 20:54:11.631793 22368 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:11.631836 22368 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:11.631882 22368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:11.632400 22368 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45105 } }
I20260619 20:54:11.632514 22368 leader_election.cc:302] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21b15ffbc7e45d3a9cbb20291a55025; no voters:
I20260619 20:54:11.632678 22368 leader_election.cc:288] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:11.632758 22373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:11.632939 22368 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:11.632956 22373 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [term 1 LEADER]: Becoming Leader. State: Replica: b21b15ffbc7e45d3a9cbb20291a55025, State: Running, Role: LEADER
I20260619 20:54:11.633422 22373 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45105 } }
I20260619 20:54:11.634027 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45105 } } }
I20260619 20:54:11.634121 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:11.634713 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:11.634362 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21b15ffbc7e45d3a9cbb20291a55025. Latest consensus state: current_term: 1 leader_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21b15ffbc7e45d3a9cbb20291a55025" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45105 } } }
I20260619 20:54:11.635062 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:11.635439 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:11.624836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44775 (local address 127.0.98.62:45105)
0619 20:54:11.625027 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:11.625036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:11.625074 (+ 38us) server_negotiation.cc:407] Connection header received
0619 20:54:11.625782 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:11.625791 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:11.625860 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:11.626004 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:11.626582 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:11.627928 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:11.628687 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:11.629225 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:11.629352 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:11.631577 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:11.631580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:11.631581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:11.631598 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:11.633700 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:11.634036 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:11.634039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:11.634040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:11.634139 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:11.634385 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:11.634386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:11.634386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:11.634716 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:11.634842 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:11.635207 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 20:54:11.635305 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260619 20:54:11.635528 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:11.636375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:11.637138 22386 catalog_manager.cc:1347] Generated new cluster ID: 3fc3856853db47b890e470959af63768
I20260619 20:54:11.637190 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:11.643729 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:11.637839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57119 (local address 127.0.98.62:45105)
0619 20:54:11.637876 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:11.637878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:11.637989 (+ 111us) server_negotiation.cc:407] Connection header received
0619 20:54:11.638056 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:11.638057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:11.638066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:11.638119 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:11.638617 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:11.639081 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:11.639786 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:11.640018 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:11.640104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:11.641246 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:11.641248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:11.641248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:11.641255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:11.642657 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:11.642998 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:11.642999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:11.642999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:11.643052 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:11.643264 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:11.643265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:11.643265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:11.643372 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:11.643468 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:11.643567 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:54:11.643639 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:11.650931 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:11.651626 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:11.662349 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21b15ffbc7e45d3a9cbb20291a55025: Generated new TSK 0
I20260619 20:54:11.662552 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:11.670753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22300
2026-06-19T20:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:11 dist-test-slave-0smx krb5kdc[22399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:11 dist-test-slave-0smx krb5kdc[22399](info): set up 2 sockets
Jun 19 20:54:11 dist-test-slave-0smx krb5kdc[22399](info): commencing operation
krb5kdc: starting...
W20260619 20:54:12.701362 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:12 dist-test-slave-0smx krb5kdc[22399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:12.816263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:12.894132 22412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:12.894351 22412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:12.894389 22412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:12.895509 22412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:12.895574 22412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:12.895597 22412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:12.895618 22412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:12.895645 22412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:12.897135 22412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:12.897476 22412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:12.897737 22412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:12.902308 22417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:12.902287 22418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:12.902299 22420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:12.902426 22412 server_base.cc:1034] running on GCE node
I20260619 20:54:12.902755 22412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:12.903064 22412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:12.904227 22412 hybrid_clock.cc:648] HybridClock initialized: now 1781902452904204 us; error 35 us; skew 500 ppm
Jun 19 20:54:12 dist-test-slave-0smx krb5kdc[22399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:12.906597 22412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:12.907037 22412 webserver.cc:460] Webserver started at http://127.0.98.62:33693/ using document root <none> and password file <none>
I20260619 20:54:12.907285 22412 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:12.907352 22412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:12.907500 22412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:12.908517 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4"
format_stamp: "Formatted at 2026-06-19 20:54:12 on dist-test-slave-0smx"
server_key: "b081e74001dc6fc8d8d6ef5bfeea1676"
server_key_iv: "1be812225fce391fe824d4c34fc782b3"
server_key_version: "encryptionkey@0"
I20260619 20:54:12.908870 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4"
format_stamp: "Formatted at 2026-06-19 20:54:12 on dist-test-slave-0smx"
server_key: "b081e74001dc6fc8d8d6ef5bfeea1676"
server_key_iv: "1be812225fce391fe824d4c34fc782b3"
server_key_version: "encryptionkey@0"
I20260619 20:54:12.910211 22412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:54:12.910916 22427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:12.911077 22412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:12.911172 22412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4"
format_stamp: "Formatted at 2026-06-19 20:54:12 on dist-test-slave-0smx"
server_key: "b081e74001dc6fc8d8d6ef5bfeea1676"
server_key_iv: "1be812225fce391fe824d4c34fc782b3"
server_key_version: "encryptionkey@0"
I20260619 20:54:12.911252 22412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:12.925988 22412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:12.927759 22412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:12.927922 22412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:12.931857 22412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41669
I20260619 20:54:12.931905 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41669 every 8 connection(s)
I20260619 20:54:12.932243 22412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:12.933039 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:12.935166 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4: Bootstrap starting.
I20260619 20:54:12.935708 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:12.935979 22480 log.cc:824] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:12.936607 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4: No bootstrap required, opened a new log
I20260619 20:54:12.938148 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260619 20:54:12.938274 22480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:12.938297 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71bdedcafd4a84b3a9e8c2d53177f4, State: Initialized, Role: FOLLOWER
I20260619 20:54:12.938475 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260619 20:54:12.938560 22480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:12.938602 22480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:12.938652 22480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:12.939139 22480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260619 20:54:12.939245 22480 leader_election.cc:302] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d71bdedcafd4a84b3a9e8c2d53177f4; no voters:
I20260619 20:54:12.939452 22480 leader_election.cc:288] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:12.939493 22483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:12.939658 22483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2d71bdedcafd4a84b3a9e8c2d53177f4, State: Running, Role: LEADER
I20260619 20:54:12.939697 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:12.939771 22483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260619 20:54:12.940173 22484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } } }
I20260619 20:54:12.940187 22485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d71bdedcafd4a84b3a9e8c2d53177f4. Latest consensus state: current_term: 1 leader_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71bdedcafd4a84b3a9e8c2d53177f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } } }
I20260619 20:54:12.940343 22485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:12.940327 22484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:12.940424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22412
I20260619 20:54:12.940518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:12.940713 22487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:12.940737 392 external_mini_cluster.cc:1428] Setting key 9aabcd6a2bf645e2f2fcc571d4c03c5c
I20260619 20:54:12.941233 22487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:12.942983 22487 catalog_manager.cc:1347] Generated new cluster ID: 63a8914397924e42b41fdb39f5e86e4d
I20260619 20:54:12.943035 22487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:12.947903 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:12.941190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53023 (local address 127.0.98.62:41669)
0619 20:54:12.941400 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:12.941409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:12.942358 (+ 949us) server_negotiation.cc:407] Connection header received
0619 20:54:12.942506 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:12.942513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:12.942568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:12.942709 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:12.942938 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:12.944312 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:12.945501 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:12.946122 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:12.946202 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:12.947034 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:12.947037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:12.947038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:12.947270 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:12.947309 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:12.947324 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:12.947380 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:12.947623 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 20:54:12.947681 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260619 20:54:12.948812 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:12.953346 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:12.950254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56751 (local address 127.0.98.62:41669)
0619 20:54:12.950287 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:12.950289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:12.950298 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:54:12.950335 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:12.950336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:12.950345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:12.950399 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:12.950496 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:12.951186 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:12.952125 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:12.952409 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:12.952481 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:12.953101 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:12.953102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:12.953102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:12.953122 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:12.953134 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:12.953139 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:12.953195 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:12.953246 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:54:12.953265 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:12.971807 22487 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:12.972291 22487 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:12.976765 22487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d71bdedcafd4a84b3a9e8c2d53177f4: Generated new TSK 0
I20260619 20:54:12.976933 22487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:12.979336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22412
2026-06-19T20:54:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:13 dist-test-slave-0smx krb5kdc[22511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:13 dist-test-slave-0smx krb5kdc[22511](info): set up 2 sockets
Jun 19 20:54:13 dist-test-slave-0smx krb5kdc[22511](info): commencing operation
krb5kdc: starting...
W20260619 20:54:14.005539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:14 dist-test-slave-0smx krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:14.121968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:14.201097 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:14.201313 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:14.201337 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:14.203058 22524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:14.203110 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:14.203125 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:14.203138 22524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:14.203171 22524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:14.204655 22524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:14.204926 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:14.205164 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:54:14.210104 22524 server_base.cc:1034] running on GCE node
W20260619 20:54:14.210075 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:14.210144 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:14.210067 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:14.210558 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:14.210863 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:14.212026 22524 hybrid_clock.cc:648] HybridClock initialized: now 1781902454212016 us; error 46 us; skew 500 ppm
Jun 19 20:54:14 dist-test-slave-0smx krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:14.214483 22524 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:14.214903 22524 webserver.cc:460] Webserver started at http://127.0.98.62:42225/ using document root <none> and password file <none>
I20260619 20:54:14.215138 22524 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:14.215210 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:14.215359 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:14.216413 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "ff517e782d074d6fa72c76cdac24b9b3"
format_stamp: "Formatted at 2026-06-19 20:54:14 on dist-test-slave-0smx"
server_key: "780f28af5ed96cdc75ed094566e0cdd6"
server_key_iv: "76eb9c93ab7427e3032b31948f8e9aea"
server_key_version: "encryptionkey@0"
I20260619 20:54:14.216773 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff517e782d074d6fa72c76cdac24b9b3"
format_stamp: "Formatted at 2026-06-19 20:54:14 on dist-test-slave-0smx"
server_key: "780f28af5ed96cdc75ed094566e0cdd6"
server_key_iv: "76eb9c93ab7427e3032b31948f8e9aea"
server_key_version: "encryptionkey@0"
I20260619 20:54:14.218134 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:54:14.218902 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:14.219106 22524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:54:14.219202 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "ff517e782d074d6fa72c76cdac24b9b3"
format_stamp: "Formatted at 2026-06-19 20:54:14 on dist-test-slave-0smx"
server_key: "780f28af5ed96cdc75ed094566e0cdd6"
server_key_iv: "76eb9c93ab7427e3032b31948f8e9aea"
server_key_version: "encryptionkey@0"
I20260619 20:54:14.219283 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:14.222577 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:14.224583 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:14.224740 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:14.228382 22524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34245
I20260619 20:54:14.228413 22591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34245 every 8 connection(s)
I20260619 20:54:14.228746 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:14.229779 22592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:14.231962 22592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3: Bootstrap starting.
I20260619 20:54:14.232508 22592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:14.232785 22592 log.cc:824] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:14.233485 22592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3: No bootstrap required, opened a new log
I20260619 20:54:14.234999 22592 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff517e782d074d6fa72c76cdac24b9b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260619 20:54:14.235131 22592 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:54:14.235181 22592 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff517e782d074d6fa72c76cdac24b9b3, State: Initialized, Role: FOLLOWER
I20260619 20:54:14.235342 22592 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff517e782d074d6fa72c76cdac24b9b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260619 20:54:14.235405 22592 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:54:14.235426 22592 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:54:14.235491 22592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:14.236017 22592 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff517e782d074d6fa72c76cdac24b9b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260619 20:54:14.236130 22592 leader_election.cc:302] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff517e782d074d6fa72c76cdac24b9b3; no voters:
I20260619 20:54:14.236304 22592 leader_election.cc:288] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:54:14.236310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22524
I20260619 20:54:14.236328 22595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:14.236418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:14.236450 22595 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [term 1 LEADER]: Becoming Leader. State: Replica: ff517e782d074d6fa72c76cdac24b9b3, State: Running, Role: LEADER
I20260619 20:54:14.236559 22592 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:14.236600 392 external_mini_cluster.cc:1428] Setting key 5225028574f346f65fc7236f4ccae7fc
I20260619 20:54:14.236586 22595 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff517e782d074d6fa72c76cdac24b9b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260619 20:54:14.237046 22595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff517e782d074d6fa72c76cdac24b9b3. Latest consensus state: current_term: 1 leader_uuid: "ff517e782d074d6fa72c76cdac24b9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff517e782d074d6fa72c76cdac24b9b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } } }
I20260619 20:54:14.237143 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff517e782d074d6fa72c76cdac24b9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff517e782d074d6fa72c76cdac24b9b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } } }
I20260619 20:54:14.237200 22595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:14.237231 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:14.237440 22601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:14.238018 22601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:14.239164 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:14.237211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34653 (local address 127.0.98.62:34245)
0619 20:54:14.237665 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:14.237674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:14.237715 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:54:14.237827 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:14.237835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:14.237868 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:14.238309 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:14.238707 (+ 398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:14.238710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:14.238711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:14.238714 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:14.238757 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:14.238768 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:14.238841 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:14.238912 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 20:54:14.239037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":119,"threads_started":1}
I20260619 20:54:14.239833 22601 catalog_manager.cc:1347] Generated new cluster ID: 256e9b27b3e24210bced12dd84e861d5
I20260619 20:54:14.239887 22601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:14.246145 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:14.247166 22601 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:14.247349 22601 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:14.248780 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:14.247602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35889 (local address 127.0.98.62:34245)
0619 20:54:14.247640 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:14.247642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:14.247768 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:54:14.247843 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:14.247844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:14.247849 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:14.247904 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:14.248527 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:14.248528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:14.248529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:54:14.248533 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:14.248548 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:54:14.248552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:14.248597 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:14.248659 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:54:14.248675 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:54:14.261514 22601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff517e782d074d6fa72c76cdac24b9b3: Generated new TSK 0
I20260619 20:54:14.261740 22601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:14.264313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22524
2026-06-19T20:54:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1284 ms)
[----------] 18 tests from AuthTokenIssuingTest (23435 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:14 dist-test-slave-0smx krb5kdc[22623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:14 dist-test-slave-0smx krb5kdc[22623](info): set up 2 sockets
Jun 19 20:54:14 dist-test-slave-0smx krb5kdc[22623](info): commencing operation
krb5kdc: starting...
W20260619 20:54:15.293491 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:15.409415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35793
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35793,127.0.98.61:43989,127.0.98.60:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:15.490983 22636 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:54:15.491189 22636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:15.491206 22636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:15.491225 22636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:15.492343 22636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:15.492383 22636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:15.492396 22636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:15.492408 22636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:15.492420 22636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:15.494001 22636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35793,127.0.98.61:43989,127.0.98.60:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:15.494248 22636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:15.494475 22636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:15.499294 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:15.499284 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:15.499409 22636 server_base.cc:1034] running on GCE node
W20260619 20:54:15.499513 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:15.499837 22636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:15.500150 22636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:15.501351 22636 hybrid_clock.cc:648] HybridClock initialized: now 1781902455501325 us; error 29 us; skew 500 ppm
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:15.506910 22636 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:15.507388 22636 webserver.cc:460] Webserver started at http://127.0.98.62:37773/ using document root <none> and password file <none>
I20260619 20:54:15.507620 22636 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:15.507733 22636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:15.507881 22636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:15.508944 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "9d15d2e558de4b80956638184c2eb608"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "5c2cf78afaaba9f7194c9702dae422cc"
server_key_iv: "0bd793bd4ef3f9650468b86aedd927ea"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.509351 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d15d2e558de4b80956638184c2eb608"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "5c2cf78afaaba9f7194c9702dae422cc"
server_key_iv: "0bd793bd4ef3f9650468b86aedd927ea"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.510947 22636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:54:15.511670 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:15.511837 22636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:15.511937 22636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "9d15d2e558de4b80956638184c2eb608"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "5c2cf78afaaba9f7194c9702dae422cc"
server_key_iv: "0bd793bd4ef3f9650468b86aedd927ea"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.512020 22636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:15.521883 22636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:15.523602 22636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:15.523761 22636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:15.527815 22636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35793
I20260619 20:54:15.527844 22703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35793 every 8 connection(s)
I20260619 20:54:15.528194 22636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:15.529160 22704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:15.530651 22704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:15.533735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22636
I20260619 20:54:15.533846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:15.534019 392 external_mini_cluster.cc:1428] Setting key 7606dda0d08183dd3366bd28f0ce08e6
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:15.540700 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.531239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35793 (local address 127.0.0.1:59546)
0619 20:54:15.531409 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.531450 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.531886 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.532183 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.532185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.532418 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.532562 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.532565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.533596 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.533597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.534739 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.534742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.535245 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.535246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.535335 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.535865 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.535872 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.537203 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.539299 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.539305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.539307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.539488 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.539771 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.539771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.539771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.539843 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.540297 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.540301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.540350 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.540434 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.540552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260619 20:54:15.540774 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.531292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.98.62:35793)
0619 20:54:15.531457 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.531461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.531526 (+ 65us) server_negotiation.cc:407] Connection header received
0619 20:54:15.532040 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.532042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.532066 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.532157 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.532662 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.533476 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.534829 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.535139 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.535223 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.537329 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.537338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.537339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.537346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.539199 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.539575 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.539579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.539579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.539688 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.539925 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.539926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.539926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.540213 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.540318 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.540493 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.540626 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260619 20:54:15.541550 22704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:15.541910 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.541789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49722)
0619 20:54:15.541839 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.541886 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260619 20:54:15.542075 22652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
I20260619 20:54:15.542491 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.542403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49730)
0619 20:54:15.542449 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.542475 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:54:15.542579 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43989: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 20:54:15.557958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43989
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35793,127.0.98.61:43989,127.0.98.60:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:54:15.591789 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } attempt: 1
I20260619 20:54:15.592422 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.592171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49740)
0619 20:54:15.592316 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.592376 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 20:54:15.592828 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.592706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49742)
0619 20:54:15.592757 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.592795 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:54:15.592955 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43989: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
W20260619 20:54:15.637655 22711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:54:15.637847 22711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:15.637866 22711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:15.637884 22711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:15.639003 22711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:15.639043 22711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:15.639057 22711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:15.639070 22711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:15.639123 22711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:15.640558 22711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35793,127.0.98.61:43989,127.0.98.60:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:15.640779 22711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:15.640990 22711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:15.645576 22719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:15.645583 22716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:15.645627 22717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:15.645722 22711 server_base.cc:1034] running on GCE node
I20260619 20:54:15.646099 22711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:15.646399 22711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:15.647539 22711 hybrid_clock.cc:648] HybridClock initialized: now 1781902455647524 us; error 36 us; skew 500 ppm
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:15.649864 22711 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 20:54:15.650319 22711 webserver.cc:460] Webserver started at http://127.0.98.61:45073/ using document root <none> and password file <none>
I20260619 20:54:15.650527 22711 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:15.650570 22711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:15.650733 22711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:15.651700 22711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data/instance:
uuid: "57dc16cca6964934883ddf729603fde1"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "c4d127e68cacec3bb34b20a9eaca68dd"
server_key_iv: "a6f0ed962dbf3b46a6a8abe9723077c0"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.652040 22711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal/instance:
uuid: "57dc16cca6964934883ddf729603fde1"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "c4d127e68cacec3bb34b20a9eaca68dd"
server_key_iv: "a6f0ed962dbf3b46a6a8abe9723077c0"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.653354 22711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:54:15.654044 22726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:15.654220 22711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:15.654311 22711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal
uuid: "57dc16cca6964934883ddf729603fde1"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "c4d127e68cacec3bb34b20a9eaca68dd"
server_key_iv: "a6f0ed962dbf3b46a6a8abe9723077c0"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.654398 22711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:15.661161 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } attempt: 2
I20260619 20:54:15.661691 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.661546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49758)
0619 20:54:15.661614 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.661658 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:54:15.662096 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.661987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49772)
0619 20:54:15.662040 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.662068 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 20:54:15.662263 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43989: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43989: connect: Connection refused (error 111)
I20260619 20:54:15.674947 22711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:15.677134 22711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:15.677296 22711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:15.681278 22711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43989
I20260619 20:54:15.681357 22778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43989 every 8 connection(s)
I20260619 20:54:15.681742 22711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:54:15.682435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22711
I20260619 20:54:15.682566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-1/wal/instance
I20260619 20:54:15.682758 392 external_mini_cluster.cc:1428] Setting key eefb0dcca686c61199610a83c0e042f7
I20260619 20:54:15.682969 22779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:15.685026 22779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:15.696851 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.685783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.98.62:35793)
0619 20:54:15.685857 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.685859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.686102 (+ 243us) server_negotiation.cc:407] Connection header received
0619 20:54:15.686824 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.686826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.686838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.686891 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.687687 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.688181 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.689162 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.689428 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.689514 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.693917 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.693919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.693919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.693928 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.695492 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.695879 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.695881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.695881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.695945 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.696170 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.696171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.696172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.696328 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.696403 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.696626 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.696726 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:54:15.696862 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.685750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35793 (local address 127.0.0.1:59550)
0619 20:54:15.685955 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.686002 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.686713 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.686972 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.686976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.687333 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.687593 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.687598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.688276 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.688277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.689056 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.689058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.689542 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.689544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.689659 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.690190 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.690198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.693825 (+ 3627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.695574 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.695595 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.695597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.695808 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.696015 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.696015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.696015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.696108 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.696395 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.696399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.696515 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.696594 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.696708 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260619 20:54:15.697602 22779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:54:15.706167 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.697929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.0.98.61:43989)
0619 20:54:15.698058 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.698061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.698075 (+ 14us) server_negotiation.cc:407] Connection header received
0619 20:54:15.698089 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.698090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.698100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.698151 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.698492 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.699174 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.700237 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.700525 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.700580 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.703610 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.703618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.703619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.703626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.705140 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.705523 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.705527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.705527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.705591 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.705742 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.705743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.705743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.705862 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.705923 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.706014 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.706097 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260619 20:54:15.706269 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.697854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49782)
0619 20:54:15.697901 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.697911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.697959 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.698183 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.698184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.698350 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.698411 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.698413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.699256 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.699258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.700140 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.700143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.700605 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.700607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.700680 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.701031 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.701036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.703525 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.705217 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.705218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.705218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.705448 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.705656 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.705656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.705657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.705700 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.705908 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.705908 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.705939 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.705982 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.706143 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":58}
I20260619 20:54:15.706830 22779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:15.707165 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.707062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39212)
0619 20:54:15.707095 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.707136 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 20:54:15.707281 22728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 20:54:15.707651 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.707568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39226)
0619 20:54:15.707608 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.707628 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 20:54:15.707733 22779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
I20260619 20:54:15.708178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45191
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35793,127.0.98.61:43989,127.0.98.60:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:54:15.753448 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } attempt: 3
I20260619 20:54:15.756932 22779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } attempt: 1
I20260619 20:54:15.757365 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.757205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39228)
0619 20:54:15.757265 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.757330 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 20:54:15.757742 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.757633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39244)
0619 20:54:15.757685 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.757717 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260619 20:54:15.757836 22779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
I20260619 20:54:15.760586 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.753896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.98.61:43989)
0619 20:54:15.753964 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.753967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.753977 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:54:15.754044 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.754045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.754057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.754127 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.754445 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.754952 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.755667 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.755891 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.755967 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.758190 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.758192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.758192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.758199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.759500 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.759826 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.759827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.759828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.759884 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.760094 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.760095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.760095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.760204 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.760281 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.760432 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.760503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:54:15.760592 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.753825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49784)
0619 20:54:15.753882 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.753901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.753974 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.754133 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.754134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.754288 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.754354 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.754355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.755044 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.755045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.755578 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.755579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.755963 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.755964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.756048 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.756555 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.756559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.758107 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.759585 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.759586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.759586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.759756 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.759969 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.759970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.759970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.760032 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.760272 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.760273 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.760319 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.760407 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.760500 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:54:15.760914 22704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:15.761234 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.761142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39252)
0619 20:54:15.761189 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.761212 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:54:15.761597 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.761536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39258)
0619 20:54:15.761562 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.761583 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 20:54:15.761693 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
W20260619 20:54:15.787827 22786 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:54:15.788023 22786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:15.788040 22786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:15.788077 22786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:15.789179 22786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:15.789222 22786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:15.789234 22786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:15.789247 22786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:15.789311 22786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:15.790741 22786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35793,127.0.98.61:43989,127.0.98.60:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:15.790989 22786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:15.791235 22786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:54:15.792824 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } attempt: 1
I20260619 20:54:15.793251 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.793106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39272)
0619 20:54:15.793168 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.793213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:54:15.793603 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.793514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39282)
0619 20:54:15.793557 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.793584 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:54:15.793694 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
W20260619 20:54:15.796219 22794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:15.796226 22792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:15.796344 22791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:15.796380 22786 server_base.cc:1034] running on GCE node
I20260619 20:54:15.796833 22786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:15.797114 22786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:15.798273 22786 hybrid_clock.cc:648] HybridClock initialized: now 1781902455798251 us; error 51 us; skew 500 ppm
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:15.800725 22786 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 20:54:15.801180 22786 webserver.cc:460] Webserver started at http://127.0.98.60:34033/ using document root <none> and password file <none>
I20260619 20:54:15.801469 22786 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:15.801539 22786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:15.801698 22786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:15.802745 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data/instance:
uuid: "b55e1bc774f3412391f10c5c09a83f73"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "ce842ac2d4e6544dfc556bdd09fba75b"
server_key_iv: "c4dd9d03c9f6d121f22ce1d35e3b0378"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.803104 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal/instance:
uuid: "b55e1bc774f3412391f10c5c09a83f73"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "ce842ac2d4e6544dfc556bdd09fba75b"
server_key_iv: "c4dd9d03c9f6d121f22ce1d35e3b0378"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.804373 22786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:15.805076 22801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:15.805246 22786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:15.805359 22786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal
uuid: "b55e1bc774f3412391f10c5c09a83f73"
format_stamp: "Formatted at 2026-06-19 20:54:15 on dist-test-slave-0smx"
server_key: "ce842ac2d4e6544dfc556bdd09fba75b"
server_key_iv: "c4dd9d03c9f6d121f22ce1d35e3b0378"
server_key_version: "encryptionkey@0"
I20260619 20:54:15.805457 22786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:15.824108 22786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:15.825986 22779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } attempt: 2
I20260619 20:54:15.826423 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.826294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39284)
0619 20:54:15.826359 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.826393 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:54:15.826695 22786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:15.826815 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.826724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39298)
0619 20:54:15.826766 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.826793 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260619 20:54:15.826862 22786 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260619 20:54:15.826910 22779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45191: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45191: connect: Connection refused (error 111)
I20260619 20:54:15.830720 22786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45191
I20260619 20:54:15.830752 22853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45191 every 8 connection(s)
I20260619 20:54:15.831104 22786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:54:15.832078 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:15.832527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 22786
I20260619 20:54:15.832640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781902166366061-392-0/minicluster-data/master-2/wal/instance
I20260619 20:54:15.832824 392 external_mini_cluster.cc:1428] Setting key e4ae00e8fecc7e67d67f41f723d18d71
I20260619 20:54:15.834187 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:54:15.842850 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.834887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.0.98.62:35793)
0619 20:54:15.834960 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.834962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.835147 (+ 185us) server_negotiation.cc:407] Connection header received
0619 20:54:15.835192 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.835193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.835202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.835241 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.835782 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.836223 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.837144 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.837371 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.837436 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.839615 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.839616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.839616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.839624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.841266 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.841763 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.841765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.841765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.841822 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.842088 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.842089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.842089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.842229 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.842294 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.842653 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.842741 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:54:15.843115 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.833387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.98.60:45191)
0619 20:54:15.833585 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.833594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.833629 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:54:15.834156 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.834165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.834224 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.834375 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.835125 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.836247 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.836991 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.837435 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.837546 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.840122 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.840130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.840131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.840143 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.841750 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.842116 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.842119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.842119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.842178 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.842387 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.842388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.842388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.842515 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.842617 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.842754 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.842846 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260619 20:54:15.843321 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.834831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35793 (local address 127.0.0.1:59556)
0619 20:54:15.835024 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.835054 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.835138 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.835265 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.835268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.835586 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.835701 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.835707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.836311 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.836312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.837067 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.837068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.837470 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.837471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.837546 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.838168 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.838177 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.839513 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.841397 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.841404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.841406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.841673 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.841895 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.841896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.841896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.842013 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.842322 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.842329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.842484 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.843066 (+ 582us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.843207 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260619 20:54:15.843781 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:15.850085 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.844049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43989 (local address 127.0.0.1:49796)
0619 20:54:15.844079 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.844089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.844122 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.844259 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.844259 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.844406 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.844441 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.844443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.845063 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.845064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.845593 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.845595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.845981 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.845982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.846042 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.846366 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.846370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.847546 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.849096 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.849097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.849098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.849280 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.849514 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.849515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.849515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.849575 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.849834 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.849835 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.849879 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.849940 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.850024 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 20:54:15.850149 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.844103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.0.98.61:43989)
0619 20:54:15.844142 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.844144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.844153 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:54:15.844175 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.844176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.844185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.844235 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.844517 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.844960 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.845690 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.845911 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.845986 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.847628 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.847629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.847629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.847634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.849018 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.849376 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.849377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.849378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.849442 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.849632 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.849633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.849634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.849763 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.849835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.849988 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.850065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:15.850509 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:15.851785 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.843874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.98.62:35793)
0619 20:54:15.843910 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.843912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.843958 (+ 46us) server_negotiation.cc:407] Connection header received
0619 20:54:15.844022 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.844023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.844032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.844099 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.844745 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.845196 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.845884 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.846094 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.846420 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.848737 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.848739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.848740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.848747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.850726 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.851122 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.851124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.851124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.851185 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.851375 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.851376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.851376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.851481 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.851603 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.851677 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.851729 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:54:15.857580 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.850744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39324)
0619 20:54:15.850788 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.850798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.850835 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.850965 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.850965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.851066 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.851117 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.851118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.851964 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.851965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.852514 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.852516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.853054 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.853055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.853094 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.853551 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.853556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.854570 (+ 1014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.855965 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.855966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.855967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.856192 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.856900 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.856901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.856902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.856967 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.857261 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.857262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.857314 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.857381 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.857475 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:54:15.857736 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.850791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.0.98.60:45191)
0619 20:54:15.850829 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.850830 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.850838 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:54:15.850889 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.850890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.850907 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.850954 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.851191 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.851904 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.852615 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.852968 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.853042 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.854641 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.854643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.854643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.854650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.855886 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.856776 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.856777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.856778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.856833 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.857078 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.857079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.857080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.857211 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.857269 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.857411 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.857525 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":18}
I20260619 20:54:15.859021 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: Bootstrap starting.
I20260619 20:54:15.859879 22854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:15.859908 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.852588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.0.98.61:43989)
0619 20:54:15.852629 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.852631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.852654 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:54:15.852672 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.852675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.852687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.852729 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.853336 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.853780 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.854444 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.854688 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.854755 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.857194 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.857195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.857195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.857200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.858774 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.859149 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.859150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.859150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.859206 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.859403 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.859408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.859409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.859530 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.859699 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.859734 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.859805 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:54:15.860235 22854 log.cc:824] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:15.861106 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: No bootstrap required, opened a new log
I20260619 20:54:15.861450 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:15.863010 22854 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.863137 22854 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55e1bc774f3412391f10c5c09a83f73, State: Initialized, Role: FOLLOWER
I20260619 20:54:15.863219 22854 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.862635 22678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9d15d2e558de4b80956638184c2eb608"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59570
I20260619 20:54:15.863516 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.863581 22854 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:15.863637 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:54:15.865410 22871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:54:15.865468 22871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:54:15.865685 22786 master_runner.cc:186] Error getting master registration for 127.0.98.62:35793: OK, instance_id { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" instance_seqno: 1781902455526679 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:54:15.867635 22786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:54:15.868813 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } attempt: 2
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:54:15.875794 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.869127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.98.60:45191)
0619 20:54:15.869161 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.869163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.869171 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:54:15.869233 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.869233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.869241 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.869287 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.869620 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.870069 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.870750 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.870949 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.871025 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.873041 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.873042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.873043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.873050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.874690 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.875018 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.875019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.875020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.875070 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.875268 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.875269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.875269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.875400 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.875487 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.875639 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.875707 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:54:15.875797 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.869075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39338)
0619 20:54:15.869124 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.869134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.869169 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.869316 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.869317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.869485 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.869529 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.869531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.870164 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.870166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.870656 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.870658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.871034 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.871035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.871094 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.871508 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.871513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.872940 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.874758 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.874759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.874759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.874932 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.875139 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.875139 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.875140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.875196 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.875482 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.875483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.875528 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.875614 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.875707 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:54:15.877238 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608: Bootstrap starting.
I20260619 20:54:15.877873 22704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:15.878153 22704 log.cc:824] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:15.879014 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608: No bootstrap required, opened a new log
I20260619 20:54:15.880038 22704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.880177 22704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d15d2e558de4b80956638184c2eb608, State: Initialized, Role: FOLLOWER
I20260619 20:54:15.880273 22704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.880618 22704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:15.880687 22678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 20:54:15.880754 22678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:15.880571 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.881444 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:15.881632 22678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.882160 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d15d2e558de4b80956638184c2eb608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b55e1bc774f3412391f10c5c09a83f73"
I20260619 20:54:15.882294 22828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:15.883175 22828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d15d2e558de4b80956638184c2eb608 in term 1.
I20260619 20:54:15.883730 22653 leader_election.cc:302] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d15d2e558de4b80956638184c2eb608, b55e1bc774f3412391f10c5c09a83f73; no voters:
I20260619 20:54:15.883842 22874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:15.883849 22678 leader_election.cc:288] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [CANDIDATE]: Term 1 election: Requested vote from peers 57dc16cca6964934883ddf729603fde1 (127.0.98.61:43989), b55e1bc774f3412391f10c5c09a83f73 (127.0.98.60:45191)
I20260619 20:54:15.884153 22874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [term 1 LEADER]: Becoming Leader. State: Replica: 9d15d2e558de4b80956638184c2eb608, State: Running, Role: LEADER
I20260619 20:54:15.884251 22874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.884620 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d15d2e558de4b80956638184c2eb608. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.884681 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: This master's current role is: LEADER
W20260619 20:54:15.885272 22885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:54:15.885337 22885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:54:15.883754 22636 master_runner.cc:186] Error getting master registration for 127.0.98.61:43989: OK, instance_id { permanent_uuid: "57dc16cca6964934883ddf729603fde1" instance_seqno: 1781902455680129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:54:15.886647 22636 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:54:15.889443 22891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:15.889637 22891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:15.891433 22828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [term 1 FOLLOWER]: Refusing update from remote peer 9d15d2e558de4b80956638184c2eb608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:54:15.891789 22886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:54:15.893050 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d15d2e558de4b80956638184c2eb608. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.893142 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:15.894547 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.894660 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:15.895064 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.895150 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:15.895588 22897 mvcc.cc:205] Tried to move back new op lower bound from 7298672459328196608 to 7298672459302313984. Current Snapshot: MvccSnapshot[applied={T|T < 7298672459328196608}]
I20260619 20:54:15.896425 22898 mvcc.cc:205] Tried to move back new op lower bound from 7298672459328196608 to 7298672459302313984. Current Snapshot: MvccSnapshot[applied={T|T < 7298672459328196608}]
I20260619 20:54:15.898475 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.885231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.0.98.60:45191)
0619 20:54:15.885439 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.885441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.889492 (+ 4051us) server_negotiation.cc:407] Connection header received
0619 20:54:15.889587 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.889588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.889600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.889749 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.890619 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.891312 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.892475 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.892737 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.892800 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.894576 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.894577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.894577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.894583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.896595 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.896978 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.896979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.896979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.897063 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.897338 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.897339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.897340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.897619 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.897775 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.898151 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.898326 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 20:54:15.905458 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.885840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.0.98.61:43989)
0619 20:54:15.885879 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.885880 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.897424 (+ 11544us) server_negotiation.cc:407] Connection header received
0619 20:54:15.897525 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.897526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.897537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.897572 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.898298 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.899021 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.900013 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.900343 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.900400 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.902089 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.902090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.902090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.902097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.903778 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.904155 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.904156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.904156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.904206 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.904431 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.904432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.904432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.904553 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.905153 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.905198 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.905278 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:54:15.905812 22891 catalog_manager.cc:1347] Generated new cluster ID: 4eb07b9c9cfd4ae39c794c021f65a32b
I20260619 20:54:15.905861 22891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:15.909858 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.885624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.0.98.62:35793)
0619 20:54:15.885674 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.885676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.897432 (+ 11756us) server_negotiation.cc:407] Connection header received
0619 20:54:15.897529 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.897530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.897540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.897574 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.898244 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.898978 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.900208 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.900494 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.900549 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.907130 (+ 6581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.907131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.907131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.907138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.908759 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.909127 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.909127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.909128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.909178 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.909371 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.909372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.909372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.909482 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.909563 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.909692 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.909776 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:54:15.918016 22779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } attempt: 3
I20260619 20:54:15.918960 22891 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:15.920163 22891 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 20:54:15 dist-test-slave-0smx krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:54:15.928946 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.918504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45191 (local address 127.0.0.1:39354)
0619 20:54:15.918550 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:15.918561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:15.918598 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:15.918822 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:15.918823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:15.919004 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:15.919050 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.919052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.921140 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.921141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.921955 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:15.921965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.922453 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.922454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:15.922505 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.922923 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:15.922929 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:15.924385 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:15.927775 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.927776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.927776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.927985 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.928256 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:15.928256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:15.928257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:15.928320 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:15.928668 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:15.928669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:15.928719 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:54:15.928760 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 20:54:15.928848 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:54:15.929688 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:15.918543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.0.98.60:45191)
0619 20:54:15.918591 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.918593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.918601 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:54:15.918657 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.918658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.918667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.918725 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.920292 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.921015 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.922060 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:15.922358 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:15.922407 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:15.924484 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:15.924486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:15.924486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:15.924493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:15.927672 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.928066 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.928067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.928067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.928166 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:15.928390 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:15.928390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:15.928390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:15.928582 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:15.929363 (+ 781us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:15.929429 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 20:54:15.929551 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:54:15.931381 22779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: Bootstrap starting.
I20260619 20:54:15.932113 22779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:15.932423 22779 log.cc:824] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:15.937635 22891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608: Generated new TSK 0
I20260619 20:54:15.938086 22891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:15.941948 22779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: No bootstrap required, opened a new log
I20260619 20:54:15.944010 22779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.944135 22779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dc16cca6964934883ddf729603fde1, State: Initialized, Role: FOLLOWER
I20260619 20:54:15.944207 22779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } }
I20260619 20:54:15.944521 22779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:15.944612 22752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:15.945659 22752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [term 1 FOLLOWER]: Refusing update from remote peer 9d15d2e558de4b80956638184c2eb608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:54:15.946341 22874 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:54:15.946699 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d15d2e558de4b80956638184c2eb608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "57dc16cca6964934883ddf729603fde1"
I20260619 20:54:15.947561 22753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d15d2e558de4b80956638184c2eb608 in term 1.
I20260619 20:54:15.947638 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.947736 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:54:15.951107 22914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:54:15.951164 22914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:54:15.951753 22916 mvcc.cc:205] Tried to move back new op lower bound from 7298672459516911616 to 7298672459302313984. Current Snapshot: MvccSnapshot[applied={T|T < 7298672459328196608}]
I20260619 20:54:15.952330 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d15d2e558de4b80956638184c2eb608. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.952415 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:15.952453 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:15.952502 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:54:15.968240 22857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.966735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.98.60:45191)
0619 20:54:15.966777 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.966779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.967607 (+ 828us) server_negotiation.cc:407] Connection header received
0619 20:54:15.967706 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.967707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.967717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.967746 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.968174 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39360: BlockingRecv error: recv got EOF from 127.0.0.1:39360 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:54:15.968948 22707 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.967086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.0.98.62:35793)
0619 20:54:15.967123 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.967124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.968462 (+ 1338us) server_negotiation.cc:407] Connection header received
0619 20:54:15.968555 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.968556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.968566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.968595 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.968893 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59578: BlockingRecv error: recv got EOF from 127.0.0.1:59578 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:54:15.969647 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.967371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.0.98.61:43989)
0619 20:54:15.967407 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.967409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.969166 (+ 1757us) server_negotiation.cc:407] Connection header received
0619 20:54:15.969258 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.969259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.969269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.969316 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.969591 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49836: BlockingRecv error: recv got EOF from 127.0.0.1:49836 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:54:15.971863 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.971190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.98.61:43989)
0619 20:54:15.971222 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.971224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.971382 (+ 158us) server_negotiation.cc:407] Connection header received
0619 20:54:15.971482 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.971483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.971498 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.971529 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.971820 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49852: BlockingRecv error: recv got EOF from 127.0.0.1:49852 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:54:15.974762 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.974089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.0.98.61:43989)
0619 20:54:15.974120 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.974122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.974310 (+ 188us) server_negotiation.cc:407] Connection header received
0619 20:54:15.974400 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.974401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.974410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.974437 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.974721 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49858: BlockingRecv error: recv got EOF from 127.0.0.1:49858 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:54:15.978566 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.977894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.0.98.61:43989)
0619 20:54:15.977925 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.977927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.978121 (+ 194us) server_negotiation.cc:407] Connection header received
0619 20:54:15.978210 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.978211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.978219 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.978247 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.978527 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49868: BlockingRecv error: recv got EOF from 127.0.0.1:49868 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 20:54:15.984380 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.983652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.0.98.61:43989)
0619 20:54:15.983687 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.983689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.983885 (+ 196us) server_negotiation.cc:407] Connection header received
0619 20:54:15.983987 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.983988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.983998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.984027 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.984334 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49882: BlockingRecv error: recv got EOF from 127.0.0.1:49882 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:54:15.994359 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:15.993577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.0.98.61:43989)
0619 20:54:15.993615 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:15.993617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:15.993822 (+ 205us) server_negotiation.cc:407] Connection header received
0619 20:54:15.993916 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:15.993917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:15.993926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:15.993954 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:15.994310 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49888: BlockingRecv error: recv got EOF from 127.0.0.1:49888 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:54:16.012578 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:16.011680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.0.98.61:43989)
0619 20:54:16.011723 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:16.011724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:16.011986 (+ 262us) server_negotiation.cc:407] Connection header received
0619 20:54:16.012097 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:16.012098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:16.012110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:16.012140 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:16.012515 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49898: BlockingRecv error: recv got EOF from 127.0.0.1:49898 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:54:16.047070 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:16.046092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.0.98.61:43989)
0619 20:54:16.046139 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:16.046141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:16.046444 (+ 303us) server_negotiation.cc:407] Connection header received
0619 20:54:16.046551 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:16.046552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:16.046564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:16.046595 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:16.047008 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49914: BlockingRecv error: recv got EOF from 127.0.0.1:49914 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:54:16.113354 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:16.112606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.0.98.61:43989)
0619 20:54:16.112687 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:16.112689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:16.112789 (+ 100us) server_negotiation.cc:407] Connection header received
0619 20:54:16.112834 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:16.112835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:16.112845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:16.112910 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:16.113241 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49924: BlockingRecv error: recv got EOF from 127.0.0.1:49924 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 20:54:16.243430 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:16.242664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.0.98.61:43989)
0619 20:54:16.242740 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:16.242742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:16.242836 (+ 94us) server_negotiation.cc:407] Connection header received
0619 20:54:16.242928 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:16.242929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:16.242941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:16.243016 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:16.243352 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49928: BlockingRecv error: recv got EOF from 127.0.0.1:49928 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 20:54:16.501338 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:16.500675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.98.61:43989)
0619 20:54:16.500738 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:16.500740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:16.500754 (+ 14us) server_negotiation.cc:407] Connection header received
0619 20:54:16.500816 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:16.500817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:16.500827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:16.500880 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:16.501235 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49932: BlockingRecv error: recv got EOF from 127.0.0.1:49932 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:54:16.525755 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d15d2e558de4b80956638184c2eb608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d15d2e558de4b80956638184c2eb608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35793 } } peers { permanent_uuid: "57dc16cca6964934883ddf729603fde1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43989 } } peers { permanent_uuid: "b55e1bc774f3412391f10c5c09a83f73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45191 } } }
I20260619 20:54:16.525935 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d15d2e558de4b80956638184c2eb608 [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:16.865873 22871 catalog_manager.cc:1259] Loaded cluster ID: 4eb07b9c9cfd4ae39c794c021f65a32b
I20260619 20:54:16.865945 22871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: loading cluster ID for follower catalog manager: success
I20260619 20:54:16.866773 22871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: acquiring CA information for follower catalog manager: success
I20260619 20:54:16.867105 22871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b55e1bc774f3412391f10c5c09a83f73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:54:16.951622 22914 catalog_manager.cc:1259] Loaded cluster ID: 4eb07b9c9cfd4ae39c794c021f65a32b
I20260619 20:54:16.951699 22914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: loading cluster ID for follower catalog manager: success
I20260619 20:54:16.952593 22914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: acquiring CA information for follower catalog manager: success
I20260619 20:54:16.952924 22914 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57dc16cca6964934883ddf729603fde1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:54:17.017333 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:17.014368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.98.61:43989)
0619 20:54:17.014505 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:17.014507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:17.014615 (+ 108us) server_negotiation.cc:407] Connection header received
0619 20:54:17.014721 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:17.014722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:17.014801 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:17.014904 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:17.015077 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.015647 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.016473 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.016678 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.016753 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:17.016953 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:17.017061 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:17.017147 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:17.017188 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 20:54:17.017208 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":90,"threads_started":1}
I20260619 20:54:17.021567 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:17.019237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.98.60:45191)
0619 20:54:17.019331 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:17.019333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:17.019350 (+ 17us) server_negotiation.cc:407] Connection header received
0619 20:54:17.019387 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:17.019388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:17.019426 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:17.019495 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:17.019581 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.020055 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.020809 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.021007 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.021062 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:17.021192 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:17.021286 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:17.021427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:17.021444 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 20:54:17.021466 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":49,"threads_started":1}
I20260619 20:54:17.026337 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:17.023360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.0.98.61:43989)
0619 20:54:17.023398 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:17.023400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:17.023737 (+ 337us) server_negotiation.cc:407] Connection header received
0619 20:54:17.023794 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:17.023795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:17.023807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:17.023857 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:17.023993 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.024691 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.025550 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.025828 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.025887 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:17.026068 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:17.026103 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:17.026163 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:17.026219 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 20:54:17.026241 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:54:17.026364 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:17.023232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.0.98.60:45191)
0619 20:54:17.023270 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:17.023272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:17.023448 (+ 176us) server_negotiation.cc:407] Connection header received
0619 20:54:17.023522 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:17.023522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:17.023531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:17.023588 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:17.023709 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.024296 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.025340 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.025627 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.025713 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:17.026002 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:17.026033 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:17.026086 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:17.026264 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 20:54:17.026287 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:54:17.026741 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:17.023213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.0.98.62:35793)
0619 20:54:17.023411 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:17.023413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:17.023421 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:54:17.023467 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:17.023467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:17.023520 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:17.023604 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:17.023748 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.024556 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.025588 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:17.025912 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:17.025973 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:17.026166 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:17.026266 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:17.026422 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:17.026625 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 20:54:17.026654 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":43,"threads_started":1}
I20260619 20:54:17.028216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22636
I20260619 20:54:17.033697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22711
I20260619 20:54:17.038926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 22786
2026-06-19T20:54:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2776 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:54:17 dist-test-slave-0smx krb5kdc[23006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:54:17 dist-test-slave-0smx krb5kdc[23006](info): set up 2 sockets
Jun 19 20:54:17 dist-test-slave-0smx krb5kdc[23006](info): commencing operation
krb5kdc: starting...
W20260619 20:54:18.081552 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/test-user.keytab.
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 20:54:18.191891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36115,127.0.98.61:45697,127.0.98.60:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:18.270823 23019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:54:18.271070 23019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:18.271100 23019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:18.271143 23019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:18.272238 23019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:18.272301 23019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:18.272326 23019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:18.272356 23019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:18.272383 23019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:18.273861 23019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36115,127.0.98.61:45697,127.0.98.60:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:18.274137 23019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:18.274360 23019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:18.278990 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:18.279000 23024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:18.279026 23019 server_base.cc:1034] running on GCE node
W20260619 20:54:18.279009 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:18.279476 23019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:18.279706 23019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:18.280843 23019 hybrid_clock.cc:648] HybridClock initialized: now 1781902458280822 us; error 39 us; skew 500 ppm
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:18.287297 23019 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 20:54:18.287750 23019 webserver.cc:460] Webserver started at http://127.0.98.62:40289/ using document root <none> and password file <none>
I20260619 20:54:18.287925 23019 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:18.287962 23019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:18.288050 23019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:18.289018 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data/instance:
uuid: "621a3262d9974ac2ad0d36363662a17c"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "c4becc454ce44b51db385700678f6c8f"
server_key_iv: "c09ca0a2d6ca8f70b0b13a8de2896ac5"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.289371 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal/instance:
uuid: "621a3262d9974ac2ad0d36363662a17c"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "c4becc454ce44b51db385700678f6c8f"
server_key_iv: "c09ca0a2d6ca8f70b0b13a8de2896ac5"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.290580 23019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:54:18.291213 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:18.291410 23019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:54:18.291505 23019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal
uuid: "621a3262d9974ac2ad0d36363662a17c"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "c4becc454ce44b51db385700678f6c8f"
server_key_iv: "c09ca0a2d6ca8f70b0b13a8de2896ac5"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.291586 23019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:18.319156 23019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:18.321797 23019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:18.321954 23019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:18.327456 23019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36115
I20260619 20:54:18.327492 23086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36115 every 8 connection(s)
I20260619 20:54:18.327945 23019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/data/info.pb
I20260619 20:54:18.329255 23087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:18.331789 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:18.336557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 23019
I20260619 20:54:18.336663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-0/wal/instance
I20260619 20:54:18.336858 392 external_mini_cluster.cc:1428] Setting key ee94e66f66ce617bf1127d2a4da546a5
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:18.344125 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.332625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.0.1:39794)
0619 20:54:18.332824 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.332868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.333647 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.333984 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.333987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.334378 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.334668 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.334674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.336181 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.336183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.337445 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.337449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.338144 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.338146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.338236 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.338759 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.338769 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.340676 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.342673 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.342678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.342680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.342876 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.343115 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.343116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.343116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.343203 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.343540 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.343545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.343604 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.343706 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.343930 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":85,"thread_start_us":101,"threads_started":1}
I20260619 20:54:18.344128 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.332695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.0.98.62:36115)
0619 20:54:18.332836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.332846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.332976 (+ 130us) server_negotiation.cc:407] Connection header received
0619 20:54:18.333833 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.333836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.333856 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.333952 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.334764 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.336061 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.337525 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.338023 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.338133 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.340771 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.340778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.340778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.340784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.342580 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.342969 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.342972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.342972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.343050 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.343268 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.343269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.343269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.343472 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.343560 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.343731 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.343867 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260619 20:54:18.344835 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:18.345184 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.345075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45194)
0619 20:54:18.345128 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.345166 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:54:18.345340 23037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
I20260619 20:54:18.345763 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.345621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45200)
0619 20:54:18.345671 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.345694 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 20:54:18.345867 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45697: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 20:54:18.360180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45697
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36115,127.0.98.61:45697,127.0.98.60:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:54:18.395088 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } attempt: 1
I20260619 20:54:18.395654 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.395472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45204)
0619 20:54:18.395546 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.395611 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 20:54:18.396099 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.395993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45218)
0619 20:54:18.396044 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.396075 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 20:54:18.396198 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45697: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
W20260619 20:54:18.439354 23094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:54:18.439554 23094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:18.439571 23094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:18.439591 23094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:18.440712 23094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:18.440752 23094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:18.440765 23094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:18.440778 23094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:18.440832 23094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:18.442296 23094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36115,127.0.98.61:45697,127.0.98.60:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:18.442553 23094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:18.442778 23094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:18.447708 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:18.447752 23094 server_base.cc:1034] running on GCE node
W20260619 20:54:18.447698 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:18.447803 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:18.448269 23094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:18.448657 23094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:18.449810 23094 hybrid_clock.cc:648] HybridClock initialized: now 1781902458449787 us; error 37 us; skew 500 ppm
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:18.452471 23094 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 20:54:18.452944 23094 webserver.cc:460] Webserver started at http://127.0.98.61:44793/ using document root <none> and password file <none>
I20260619 20:54:18.453425 23094 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:18.453485 23094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:18.453661 23094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:18.454736 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data/instance:
uuid: "959c9d2475cc4e4daf892e84c445aba5"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "3bd06e01d7c63ecf9c6527c72583a1e2"
server_key_iv: "63360dbc429a86ea217808a800f038e5"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.455093 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal/instance:
uuid: "959c9d2475cc4e4daf892e84c445aba5"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "3bd06e01d7c63ecf9c6527c72583a1e2"
server_key_iv: "63360dbc429a86ea217808a800f038e5"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.456359 23094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:54:18.457057 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:18.457226 23094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:54:18.457343 23094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal
uuid: "959c9d2475cc4e4daf892e84c445aba5"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "3bd06e01d7c63ecf9c6527c72583a1e2"
server_key_iv: "63360dbc429a86ea217808a800f038e5"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.457437 23094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:18.464326 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } attempt: 2
I20260619 20:54:18.464684 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.464564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45234)
0619 20:54:18.464620 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.464656 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:54:18.465022 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.464919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45236)
0619 20:54:18.464962 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.464986 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:54:18.465096 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45697: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45697: connect: Connection refused (error 111)
I20260619 20:54:18.497188 23094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:18.499085 23094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:18.499254 23094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:18.503331 23094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45697
I20260619 20:54:18.503366 23161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45697 every 8 connection(s)
I20260619 20:54:18.503722 23094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/data/info.pb
I20260619 20:54:18.504678 23162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:18.504901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 23094
I20260619 20:54:18.505005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-1/wal/instance
I20260619 20:54:18.505245 392 external_mini_cluster.cc:1428] Setting key 11fa442bfdec14e5b64f0ded0fa98bc8
I20260619 20:54:18.507066 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:18.519254 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.507778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.0.98.62:36115)
0619 20:54:18.507834 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.507836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.508028 (+ 192us) server_negotiation.cc:407] Connection header received
0619 20:54:18.508514 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.508515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.508531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.508585 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.509209 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.509750 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.510756 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.510965 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.511051 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.515340 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.515342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.515342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.515349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.517807 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.518291 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.518292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.518292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.518342 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.518637 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.518637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.518638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.518737 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.518785 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.519128 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.519190 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:54:18.519567 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.507747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.0.1:39806)
0619 20:54:18.507915 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.507949 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.508403 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.508661 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.508664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.508916 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.509113 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.509117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.509852 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.509855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.510661 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.510663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.511070 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.511071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.511194 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.511689 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.511697 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.515243 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.517893 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.517899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.517902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.518208 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.518406 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.518407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.518408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.518557 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.518802 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.518808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.518984 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.519104 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.519293 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260619 20:54:18.520238 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:54:18.527390 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.520493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45246)
0619 20:54:18.520708 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.520719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.520770 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.520929 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.520929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.521069 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.521110 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.521111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.522070 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.522071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.522620 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.522621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.523140 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.523141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.523205 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.523654 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.523658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.524803 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.526482 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.526483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.526483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.526640 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.526836 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.526837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.526837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.526889 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.527095 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.527096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.527131 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.527249 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.527328 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:54:18.527503 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.520573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.98.61:45697)
0619 20:54:18.520722 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.520727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.520768 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:54:18.520827 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.520829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.520843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.520929 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.521199 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.521976 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.522710 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.523061 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.523144 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.524874 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.524881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.524881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.524887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.526411 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.526708 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.526711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.526711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.526762 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.526931 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.526931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.526931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.527044 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.527098 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.527292 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.527386 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 20:54:18.528131 23162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:18.528443 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.528351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55806)
0619 20:54:18.528384 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.528426 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 20:54:18.528554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskH2GnNK/build/release/bin/kudu
/tmp/dist-test-taskH2GnNK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44891
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36115,127.0.98.61:45697,127.0.98.60:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:54:18.528602 23111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
I20260619 20:54:18.529636 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.529529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55808)
0619 20:54:18.529577 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.529609 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:54:18.529803 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
I20260619 20:54:18.556314 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } attempt: 3
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:54:18.563383 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.556688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45258)
0619 20:54:18.556740 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.556752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.556823 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.556971 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.556972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.557133 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.557195 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.557196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.557892 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.557893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.558481 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.558483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.558852 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.558853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.558938 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.559430 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.559435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.560689 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.562360 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.562361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.562362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.562521 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.562721 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.562722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.562722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.562777 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.563053 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.563054 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.563097 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.563181 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.563284 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 20:54:18.563383 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.556752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.0.98.61:45697)
0619 20:54:18.556822 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.556824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.556834 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:54:18.556885 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.556886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.556900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.556966 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.557279 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.557803 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.558563 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.558780 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.558855 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.560784 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.560786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.560786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.560793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.562265 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.562602 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.562603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.562603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.562658 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.562869 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.562870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.562871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.562981 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.563050 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.563205 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.563285 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:54:18.563968 23087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:18.564311 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.564206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55822)
0619 20:54:18.564262 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.564288 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:54:18.564574 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.564501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55838)
0619 20:54:18.564536 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.564555 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 20:54:18.564666 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
I20260619 20:54:18.579005 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } attempt: 1
I20260619 20:54:18.579417 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.579307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55854)
0619 20:54:18.579357 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.579392 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:54:18.579720 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.579633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55870)
0619 20:54:18.579673 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.579695 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 20:54:18.579797 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
I20260619 20:54:18.595783 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } attempt: 1
I20260619 20:54:18.596226 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.596122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55882)
0619 20:54:18.596165 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.596202 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 20:54:18.596599 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.596523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55894)
0619 20:54:18.596560 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.596585 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 20:54:18.596701 23087 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
W20260619 20:54:18.610548 23169 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:54:18.610746 23169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:54:18.610762 23169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:54:18.610811 23169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:54:18.611898 23169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:54:18.611935 23169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:54:18.611948 23169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:54:18.611986 23169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:54:18.612011 23169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:54:18.613607 23169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36115,127.0.98.61:45697,127.0.98.60:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:59:35 UTC on ccycloud.quasar-bezeai.root.comops.site
build id 4456677
I20260619 20:54:18.613850 23169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:54:18.614079 23169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:54:18.618656 23175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:54:18.618685 23174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:18.618846 23169 server_base.cc:1034] running on GCE node
W20260619 20:54:18.618657 23177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:54:18.619123 23169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:54:18.619419 23169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:54:18.620573 23169 hybrid_clock.cc:648] HybridClock initialized: now 1781902458620562 us; error 41 us; skew 500 ppm
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:54:18.622980 23169 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 20:54:18.623381 23169 webserver.cc:460] Webserver started at http://127.0.98.60:45011/ using document root <none> and password file <none>
I20260619 20:54:18.623596 23169 fs_manager.cc:359] Metadata directory not provided
I20260619 20:54:18.623641 23169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:54:18.623782 23169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:54:18.624780 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data/instance:
uuid: "e81ff22057b044e7b050b6fee2dc2dab"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "b9c477f37bc5295c24296fe57f441b92"
server_key_iv: "293e2e8d9c72da04295396e650fc33f0"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.625124 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal/instance:
uuid: "e81ff22057b044e7b050b6fee2dc2dab"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "b9c477f37bc5295c24296fe57f441b92"
server_key_iv: "293e2e8d9c72da04295396e650fc33f0"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.626485 23169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:54:18.627270 23184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:54:18.627471 23169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:54:18.627542 23169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal
uuid: "e81ff22057b044e7b050b6fee2dc2dab"
format_stamp: "Formatted at 2026-06-19 20:54:18 on dist-test-slave-0smx"
server_key: "b9c477f37bc5295c24296fe57f441b92"
server_key_iv: "293e2e8d9c72da04295396e650fc33f0"
server_key_version: "encryptionkey@0"
I20260619 20:54:18.627632 23169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:54:18.647945 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } attempt: 2
I20260619 20:54:18.648406 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.648280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55908)
0619 20:54:18.648337 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.648377 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:54:18.648799 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.648672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55910)
0619 20:54:18.648749 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.648778 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 20:54:18.648919 23162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44891: connect: Connection refused (error 111)
I20260619 20:54:18.654817 23169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:54:18.657119 23169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:54:18.657287 23169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:54:18.661372 23169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44891
I20260619 20:54:18.661418 23236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44891 every 8 connection(s)
I20260619 20:54:18.661778 23169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/data/info.pb
I20260619 20:54:18.662714 23237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:54:18.663370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskH2GnNK/build/release/bin/kudu as pid 23169
I20260619 20:54:18.663477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskH2GnNK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781902166366061-392-0/minicluster-data/master-2/wal/instance
I20260619 20:54:18.663684 392 external_mini_cluster.cc:1428] Setting key 93ee5dd951ef03760e0345cf556e31b8
I20260619 20:54:18.664750 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:54:18.671869 23087 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } attempt: 2
I20260619 20:54:18.674551 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.665538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.0.98.62:36115)
0619 20:54:18.665604 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.665606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.665900 (+ 294us) server_negotiation.cc:407] Connection header received
0619 20:54:18.665927 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.665928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.665938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.666006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.666433 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.667181 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.668424 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.668700 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.668771 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.671130 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.671131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.671132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.671139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.673024 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.673569 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.673570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.673571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.673622 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.673900 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.673901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.673902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.674034 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.674067 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.674392 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.674463 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:54:18.674574 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.665480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.0.1:39812)
0619 20:54:18.665641 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.665688 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.665767 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.666021 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.666023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.666256 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.666347 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.666351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.667295 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.667296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.668355 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.668357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.668805 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.668806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.668887 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.669513 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.669523 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.671012 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.673125 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.673133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.673135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.673508 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.673692 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.673693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.673693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.673815 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.674102 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.674107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.674228 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.674315 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.674430 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260619 20:54:18.675268 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:18.675702 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.664192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.98.60:44891)
0619 20:54:18.664369 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.664378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.664414 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:54:18.664808 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.664814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.664861 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.664968 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.665933 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.667242 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.668295 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.668706 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.668802 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.671549 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.671555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.671555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.671560 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.673431 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.673803 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.673806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.673806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.673859 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.674882 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.674882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.674883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.675030 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.675164 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.675514 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.675586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 20:54:18.680943 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.672104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55932)
0619 20:54:18.672207 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.672218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.672258 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.672711 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.672712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.672960 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.673000 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.673002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.673927 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.673928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.674596 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.674599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.675215 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.675216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.675290 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.675772 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.675777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.677195 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.680002 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.680004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.680004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.680191 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.680393 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.680393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.680393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.680440 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.680717 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.680718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.680759 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.680821 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.680884 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 20:54:18.681074 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.672208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.0.98.60:44891)
0619 20:54:18.672579 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.672581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.672590 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:54:18.672608 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.672608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.672617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.672687 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.673085 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.673855 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.674829 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.675121 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.675210 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.677274 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.677275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.677276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.677282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.679918 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.680268 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.680269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.680269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.680325 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.680497 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.680498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.680498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.680615 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.680683 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.680869 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.680937 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":95,"threads_started":1}
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 20:54:18.684163 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c: Bootstrap starting.
I20260619 20:54:18.684409 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.676460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39818 (local address 127.0.98.62:36115)
0619 20:54:18.676495 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.676497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.676528 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:54:18.676545 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.676546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.676554 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.676595 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.677162 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.677679 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.678352 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.678546 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.678633 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.681719 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.681720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.681720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.681727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.683324 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.683671 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.683672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.683672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.683722 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.683932 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.683933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.683933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.684037 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.684198 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.684235 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.684305 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:54:18.684618 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.675590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.98.61:45697)
0619 20:54:18.675660 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.675662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.676396 (+ 734us) server_negotiation.cc:407] Connection header received
0619 20:54:18.676502 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.676504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.676516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.676547 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.676915 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.677694 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.678735 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.678996 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.679029 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.681638 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.681639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.681639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.681646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.683361 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.683756 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.683757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.683758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.683815 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.684039 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.684040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.684041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.684183 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.684229 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.684430 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.684508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:54:18.684912 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.675487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45697 (local address 127.0.0.1:45262)
0619 20:54:18.676343 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.676353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.676422 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.676607 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.676607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.676763 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.676819 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.676821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.677791 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.677792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.678639 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.678642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.679116 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.679117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.679189 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.679563 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.679568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.681552 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.683467 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.683468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.683468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.683685 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.683886 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.683887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.683888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.683950 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.684280 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.684281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.684320 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.684700 (+ 380us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.684784 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840}
I20260619 20:54:18.684939 23087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:18.685249 23087 log.cc:824] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:18.685262 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:54:18.686823 23087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c: No bootstrap required, opened a new log
I20260619 20:54:18.688928 23087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.689079 23087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621a3262d9974ac2ad0d36363662a17c, State: Initialized, Role: FOLLOWER
I20260619 20:54:18.689157 23087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:54:18.689467 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.689533 23087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:18.689563 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260619 20:54:18.691324 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:54:18.692498 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:54:18.692514 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.685520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55938)
0619 20:54:18.685635 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.685642 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.685673 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.685795 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.685796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.685926 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.685960 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.685961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.686793 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.686794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.687407 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.687409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.687836 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.687837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.687893 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.688234 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.688239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.689567 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.691544 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.691545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.691545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.691747 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.691932 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.691933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.691933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.691978 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.692260 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.692261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.692292 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.692347 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.692426 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 20:54:18.692740 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.685583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.0.98.60:44891)
0619 20:54:18.685615 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.685617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.685713 (+ 96us) server_negotiation.cc:407] Connection header received
0619 20:54:18.685727 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.685727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.685734 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.685769 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.686033 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.686712 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.687484 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.687742 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.687824 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.689650 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.689651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.689651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.689658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.691460 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.691827 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.691828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.691828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.691875 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.692031 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.692032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.692032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.692182 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.692240 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.692607 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.692673 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:54:18.694065 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.685027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.98.61:45697)
0619 20:54:18.685102 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.685104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.685111 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:54:18.685127 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.685127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.685135 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.685198 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.685780 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.686401 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.687187 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.687464 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.687530 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.690384 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.690385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.690386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.690393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.692094 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.693138 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.693139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.693139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.693197 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.693448 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.693449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.693449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.693579 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.693719 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.693830 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.693902 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 20:54:18.692879 23019 master_runner.cc:186] Error getting master registration for 127.0.98.61:45697: OK, instance_id { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" instance_seqno: 1781902458502169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:54:18.695184 23019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:54:18.696281 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:54:18.696530 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: Bootstrap starting.
I20260619 20:54:18.697247 23237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:18.697244 23061 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "621a3262d9974ac2ad0d36363662a17c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39818
I20260619 20:54:18.697418 23061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 20:54:18.697474 23061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:18.697585 23237 log.cc:824] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:18.698272 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: No bootstrap required, opened a new log
I20260619 20:54:18.698293 23061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.698540 23061 leader_election.cc:288] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [CANDIDATE]: Term 1 election: Requested vote from peers 959c9d2475cc4e4daf892e84c445aba5 (127.0.98.61:45697), e81ff22057b044e7b050b6fee2dc2dab (127.0.98.60:44891)
I20260619 20:54:18.700596 23237 raft_consensus.cc:348] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.700762 23237 raft_consensus.cc:729] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81ff22057b044e7b050b6fee2dc2dab, State: Initialized, Role: FOLLOWER
I20260619 20:54:18.700855 23237 consensus_queue.cc:260] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.701175 23237 sys_catalog.cc:564] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:18.701237 23211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "621a3262d9974ac2ad0d36363662a17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e81ff22057b044e7b050b6fee2dc2dab"
I20260619 20:54:18.701416 23211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:18.702230 23211 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621a3262d9974ac2ad0d36363662a17c in term 1.
I20260619 20:54:18.701200 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.702445 23036 leader_election.cc:302] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621a3262d9974ac2ad0d36363662a17c, e81ff22057b044e7b050b6fee2dc2dab; no voters:
I20260619 20:54:18.702576 23244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:54:18.702729 23244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [term 1 LEADER]: Becoming Leader. State: Replica: 621a3262d9974ac2ad0d36363662a17c, State: Running, Role: LEADER
I20260619 20:54:18.702819 23244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.702566 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:54:18.703601 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:54:18.703657 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:54:18.706521 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 621a3262d9974ac2ad0d36363662a17c. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.706625 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:18.705862 23169 master_runner.cc:186] Error getting master registration for 127.0.98.61:45697: OK, instance_id { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" instance_seqno: 1781902458502169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:54:18.707986 23169 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:54:18.710783 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.699890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.0.98.62:36115)
0619 20:54:18.699960 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.699963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.700300 (+ 337us) server_negotiation.cc:407] Connection header received
0619 20:54:18.700358 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.700358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.700367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.700424 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.701321 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.702105 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.704233 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.704543 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.704604 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.707494 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.707495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.707496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.707503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.709143 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.709707 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.709709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.709709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.709776 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.710092 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.710093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.710094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.710238 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.710307 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.710619 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.710693 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:54:18.711979 23279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:54:18.712204 23279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:54:18.714012 23211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [term 1 FOLLOWER]: Refusing update from remote peer 621a3262d9974ac2ad0d36363662a17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:54:18.714821 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:54:18.716076 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 621a3262d9974ac2ad0d36363662a17c. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.716187 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:18.717991 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.700049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.98.61:45697)
0619 20:54:18.700156 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.700158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.700423 (+ 265us) server_negotiation.cc:407] Connection header received
0619 20:54:18.706801 (+ 6378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.706802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.706814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.706844 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.708334 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.709929 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.710964 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.711276 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.711318 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.714399 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.714400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.714400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.714406 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.716292 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.716721 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.716722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.716722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.716793 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.716999 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.717000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.717000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.717125 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.717624 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.717829 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.717901 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:54:18.723781 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.723872 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:18.724346 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.724438 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:18.724782 23281 mvcc.cc:205] Tried to move back new op lower bound from 7298672470889349120 to 7298672470847299584. Current Snapshot: MvccSnapshot[applied={T|T < 7298672470889349120}]
I20260619 20:54:18.725216 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.709518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.98.60:44891)
0619 20:54:18.709554 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.709555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.709906 (+ 351us) server_negotiation.cc:407] Connection header received
0619 20:54:18.710012 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.710013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.710022 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.710051 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.713915 (+ 3864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.714623 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.715531 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.715814 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.715857 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.722272 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.722273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.722273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.722280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.723952 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.724338 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.724339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.724339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.724449 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.724655 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.724656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.724656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.724806 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.724909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.725044 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.725130 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:54:18.726158 23279 catalog_manager.cc:1347] Generated new cluster ID: 670bcf272e354a2eb4b2e714dfb5b398
I20260619 20:54:18.726207 23279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:54:18.735796 23279 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:54:18.736501 23279 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:54:18.740049 23162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } attempt: 3
Jun 19 20:54:18 dist-test-slave-0smx krb5kdc[23006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781902458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 20:54:18.749145 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.740311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44891 (local address 127.0.0.1:55956)
0619 20:54:18.740360 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:54:18.740371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:54:18.740411 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:54:18.740581 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:54:18.740582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:54:18.740756 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:54:18.740801 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.740804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.741733 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.741734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.742586 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:54:18.742589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.743051 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.743052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:54:18.743107 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.743594 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:54:18.743600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:54:18.746365 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:54:18.748135 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.748136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.748137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.748344 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.748541 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:54:18.748541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:54:18.748542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:54:18.748592 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:54:18.748845 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:54:18.748846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:54:18.748887 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:54:18.748945 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:54:18.749034 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:54:18.749706 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.740365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.0.98.60:44891)
0619 20:54:18.740400 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.740403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.740410 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:54:18.740493 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.740494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.740504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.740571 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.740878 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.741669 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.742674 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.742973 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.743041 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.746470 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:54:18.746471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:54:18.746472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:54:18.746479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:54:18.748053 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.748426 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.748427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.748428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.748479 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:54:18.748651 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:54:18.748652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:54:18.748652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:54:18.748768 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:54:18.748822 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.749545 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.749620 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:54:18.751262 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: Bootstrap starting.
I20260619 20:54:18.751962 23162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: Neither blocks nor log segments found. Creating new log.
I20260619 20:54:18.752300 23162 log.cc:824] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:54:18.753703 23162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: No bootstrap required, opened a new log
I20260619 20:54:18.755678 23162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.755810 23162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959c9d2475cc4e4daf892e84c445aba5, State: Initialized, Role: FOLLOWER
I20260619 20:54:18.755890 23162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } }
I20260619 20:54:18.756197 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.756453 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:18.756242 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "621a3262d9974ac2ad0d36363662a17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "959c9d2475cc4e4daf892e84c445aba5"
I20260619 20:54:18.756313 23135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:54:18.756215 23162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:54:18.757781 23135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [term 1 FOLLOWER]: Refusing update from remote peer 621a3262d9974ac2ad0d36363662a17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:54:18.758082 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:54:18.758817 23279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c: Generated new TSK 0
I20260619 20:54:18.759059 23279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:54:18.759378 23136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621a3262d9974ac2ad0d36363662a17c in term 1.
I20260619 20:54:18.761008 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 621a3262d9974ac2ad0d36363662a17c. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.761089 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:18.761135 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:18.761189 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:54:18.761979 23305 catalog_manager.cc:1259] Loaded cluster ID: 670bcf272e354a2eb4b2e714dfb5b398
I20260619 20:54:18.762063 23305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: loading cluster ID for follower catalog manager: success
I20260619 20:54:18.763258 23305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: acquiring CA information for follower catalog manager: success
I20260619 20:54:18.763646 23305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 959c9d2475cc4e4daf892e84c445aba5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 20:54:18.779824 23090 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.778306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.0.98.62:36115)
0619 20:54:18.778347 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.778349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.779175 (+ 826us) server_negotiation.cc:407] Connection header received
0619 20:54:18.779279 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.779280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.779290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.779320 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.779756 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39846: BlockingRecv error: recv got EOF from 127.0.0.1:39846 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:54:18.780575 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.778624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.0.98.61:45697)
0619 20:54:18.778660 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.778662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.780054 (+ 1392us) server_negotiation.cc:407] Connection header received
0619 20:54:18.780185 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.780186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.780196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.780225 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.780519 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45288: BlockingRecv error: recv got EOF from 127.0.0.1:45288 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:54:18.781388 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.778908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.98.60:44891)
0619 20:54:18.778943 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.778945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.780767 (+ 1822us) server_negotiation.cc:407] Connection header received
0619 20:54:18.780919 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.780920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.780930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.780959 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.781314 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55964: BlockingRecv error: recv got EOF from 127.0.0.1:55964 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:54:18.786414 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.782889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.0.98.61:45697)
0619 20:54:18.782923 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.782924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.783149 (+ 225us) server_negotiation.cc:407] Connection header received
0619 20:54:18.783247 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.783248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.783280 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.783323 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.783514 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.784294 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.785466 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.785763 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.785804 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.786016 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.786157 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.786207 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.786330 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.786348 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:54:18.789582 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.788740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.98.60:44891)
0619 20:54:18.788773 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.788774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.789012 (+ 238us) server_negotiation.cc:407] Connection header received
0619 20:54:18.789128 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.789129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.789161 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.789191 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.789535 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55970: BlockingRecv error: recv got EOF from 127.0.0.1:55970 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:18.794895 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.791646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.0.98.61:45697)
0619 20:54:18.791677 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.791679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.791897 (+ 218us) server_negotiation.cc:407] Connection header received
0619 20:54:18.791981 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.791982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.791990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.792015 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.792185 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.792902 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.794070 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.794389 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.794427 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.794647 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.794680 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.794708 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.794810 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.794828 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:54:18.798063 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.797367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.0.98.60:44891)
0619 20:54:18.797400 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.797402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.797594 (+ 192us) server_negotiation.cc:407] Connection header received
0619 20:54:18.797683 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.797684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.797694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.797722 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.798019 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55986: BlockingRecv error: recv got EOF from 127.0.0.1:55986 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:18.804168 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.801136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.61:45697)
0619 20:54:18.801168 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.801169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.801461 (+ 292us) server_negotiation.cc:407] Connection header received
0619 20:54:18.801544 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.801545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.801553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.801577 (+ 24us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.801720 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.802399 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.803427 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.803693 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.803735 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.803915 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.803944 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.803969 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.804062 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.804078 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:54:18.806957 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.806288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.0.98.60:44891)
0619 20:54:18.806320 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.806321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.806497 (+ 176us) server_negotiation.cc:407] Connection header received
0619 20:54:18.806608 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.806609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.806618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.806644 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.806918 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55988: BlockingRecv error: recv got EOF from 127.0.0.1:55988 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:54:18.815202 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.811997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.0.98.61:45697)
0619 20:54:18.812031 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.812033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.812288 (+ 255us) server_negotiation.cc:407] Connection header received
0619 20:54:18.812385 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.812386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.812395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.812424 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.812587 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.813318 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.814404 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.814706 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.814746 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.814933 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.814964 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.814990 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.815086 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.815127 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:54:18.818173 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.817464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.0.98.60:44891)
0619 20:54:18.817499 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.817500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.817682 (+ 182us) server_negotiation.cc:407] Connection header received
0619 20:54:18.817770 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.817771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.817781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.817808 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.818126 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55998: BlockingRecv error: recv got EOF from 127.0.0.1:55998 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:18.830514 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.827241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.0.98.61:45697)
0619 20:54:18.827277 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.827278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.827533 (+ 255us) server_negotiation.cc:407] Connection header received
0619 20:54:18.827628 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.827629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.827638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.827665 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.827830 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.828516 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.829593 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.829876 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.829919 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.830231 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.830262 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.830289 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.830430 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.830448 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:54:18.833619 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.832812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.0.98.60:44891)
0619 20:54:18.832855 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.832857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.833048 (+ 191us) server_negotiation.cc:407] Connection header received
0619 20:54:18.833159 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.833160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.833171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.833202 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.833573 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56014: BlockingRecv error: recv got EOF from 127.0.0.1:56014 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:54:18.854560 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.850975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.98.61:45697)
0619 20:54:18.851019 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.851021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.851330 (+ 309us) server_negotiation.cc:407] Connection header received
0619 20:54:18.851431 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.851432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.851443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.851471 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.851680 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.852498 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.853705 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.853980 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.854040 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.854261 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.854315 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.854346 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.854461 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.854481 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:54:18.857852 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.856958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.0.98.60:44891)
0619 20:54:18.857001 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.857003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.857217 (+ 214us) server_negotiation.cc:407] Connection header received
0619 20:54:18.857366 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.857368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.857382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.857411 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.857799 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56026: BlockingRecv error: recv got EOF from 127.0.0.1:56026 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:54:18.893767 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.891281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.0.98.61:45697)
0619 20:54:18.891330 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.891332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.891461 (+ 129us) server_negotiation.cc:407] Connection header received
0619 20:54:18.891532 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.891533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.891544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.891649 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.891761 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.892243 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.893055 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.893247 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.893340 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.893499 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.893539 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.893599 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.893662 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.893680 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:54:18.896448 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.895763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.0.98.60:44891)
0619 20:54:18.895829 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.895831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.895887 (+ 56us) server_negotiation.cc:407] Connection header received
0619 20:54:18.895965 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.895966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.895974 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.896019 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.896392 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56042: BlockingRecv error: recv got EOF from 127.0.0.1:56042 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:54:18.963724 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:18.961495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.0.98.61:45697)
0619 20:54:18.961561 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.961563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.961712 (+ 149us) server_negotiation.cc:407] Connection header received
0619 20:54:18.961762 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.961763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.961772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.961812 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:18.961927 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.962406 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.963140 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:18.963320 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:18.963368 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:18.963505 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:18.963530 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:18.963567 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:18.963618 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 20:54:18.963633 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 20:54:18.965946 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:18.965444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.0.98.60:44891)
0619 20:54:18.965505 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:18.965507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:18.965562 (+ 55us) server_negotiation.cc:407] Connection header received
0619 20:54:18.965604 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:18.965604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:18.965613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:18.965654 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:18.965909 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56052: BlockingRecv error: recv got EOF from 127.0.0.1:56052 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:54:19.097906 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.095234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.0.98.61:45697)
0619 20:54:19.095313 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.095316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.095385 (+ 69us) server_negotiation.cc:407] Connection header received
0619 20:54:19.095463 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.095464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.095476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.095550 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.095684 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.096271 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.097144 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.097387 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.097467 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.097624 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.097665 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.097727 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.097788 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.097809 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 20:54:19.100145 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:19.099551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.0.98.60:44891)
0619 20:54:19.099618 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.099620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.099644 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:54:19.099723 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.099724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.099734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.099778 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:19.100087 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56066: BlockingRecv error: recv got EOF from 127.0.0.1:56066 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:54:19.173507 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "621a3262d9974ac2ad0d36363662a17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621a3262d9974ac2ad0d36363662a17c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } peers { permanent_uuid: "959c9d2475cc4e4daf892e84c445aba5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45697 } } peers { permanent_uuid: "e81ff22057b044e7b050b6fee2dc2dab" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44891 } } }
I20260619 20:54:19.173671 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 621a3262d9974ac2ad0d36363662a17c [sys.catalog]: This master's current role is: LEADER
I20260619 20:54:19.360044 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.357547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.0.98.61:45697)
0619 20:54:19.357618 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.357620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.357672 (+ 52us) server_negotiation.cc:407] Connection header received
0619 20:54:19.357721 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.357722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.357731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.357789 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.357945 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.358608 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.359434 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.359627 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.359678 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.359811 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.359837 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.359874 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.359926 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.359940 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 20:54:19.362255 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:54:19.361739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.98.60:44891)
0619 20:54:19.361791 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.361793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.361802 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:54:19.361865 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.361865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.361874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.361916 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:54:19.362185 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56074: BlockingRecv error: recv got EOF from 127.0.0.1:56074 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:54:19.704066 23275 catalog_manager.cc:1259] Loaded cluster ID: 670bcf272e354a2eb4b2e714dfb5b398
I20260619 20:54:19.704159 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: loading cluster ID for follower catalog manager: success
I20260619 20:54:19.704948 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: acquiring CA information for follower catalog manager: success
I20260619 20:54:19.705262 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e81ff22057b044e7b050b6fee2dc2dab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:54:19.877730 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.875440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.0.98.61:45697)
0619 20:54:19.875555 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.875557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.875629 (+ 72us) server_negotiation.cc:407] Connection header received
0619 20:54:19.875673 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.875674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.875683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.875774 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.875883 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.876359 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.877075 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.877267 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.877390 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.877517 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.877545 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.877596 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.877646 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.877665 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260619 20:54:19.881743 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.879287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.0.98.60:44891)
0619 20:54:19.879385 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.879387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.879396 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:54:19.879454 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.879455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.879539 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.879634 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.879727 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.880219 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.880935 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.881156 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.881228 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.881401 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.881528 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.881616 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.881637 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.881654 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
I20260619 20:54:19.886060 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.883478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.0.98.61:45697)
0619 20:54:19.883513 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.883515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.883621 (+ 106us) server_negotiation.cc:407] Connection header received
0619 20:54:19.883706 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.883707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.883716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.883758 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.883912 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.884603 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.885336 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.885609 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.885670 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.885799 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.885828 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.885877 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.885964 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.885983 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:54:19.886313 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.883288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.0.98.60:44891)
0619 20:54:19.883322 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.883323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.883452 (+ 129us) server_negotiation.cc:407] Connection header received
0619 20:54:19.883781 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.883782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.883799 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.883856 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.884000 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.884454 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.885512 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.885815 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.885880 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.886040 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.886074 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.886103 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.886208 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.886227 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:54:19.886610 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:54:19.883282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.98.62:36115)
0619 20:54:19.883510 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 20:54:19.883512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:54:19.883519 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:54:19.883567 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:54:19.883568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:54:19.883603 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:54:19.883666 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:54:19.883791 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.884584 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.885574 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:54:19.885880 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:54:19.886084 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:54:19.886156 (+ 72us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:54:19.886255 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:54:19.886479 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 20:54:19.886499 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 20:54:19.886524 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":42,"threads_started":1}
I20260619 20:54:19.887988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 23019
I20260619 20:54:19.893615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 23094
I20260619 20:54:19.898219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskH2GnNK/build/release/bin/kudu with pid 23169
2026-06-19T20:54:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2859 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5635 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (293538 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS